AI Collection Sorting vs Manual: A 2-Week A/B Test

On June 17, Shopify quietly rolled out Campaign Autopilot.

I have been digging into what this means for merchants.

My Experience

The challenge is that there is no migration path for non-technical store owners. Here is the thing: Storefront Agent will affect how your checkout works. I tried this on my store and the results were mixed.

• I switched over the weekend and found several issues.
• Horizon themes broke and nobody noticed.
• According to merchant communities, the migration is not going smoothly.
• I spent a weekend testing what was happening.
• The key point: AI search will affect how your checkout works.

The Impact on Stores

Campaign Autopilot deprecated and nobody noticed. According to Shopify changelog, confusion is common. According to support tickets, confusion is common.

The issue is that the documentation assumes you are a developer. The situation is that there is no migration path for non-technical store owners. Horizon themes updated without warning. According to the developer forums, confusion is common.

Smart Email Delivery updated quietly. The challenge is that there is no migration path for non-technical store owners. I spent hours figuring out what was happening.

Who Is Affected

Smart Email Delivery deprecated without warning. I tested this last week and discovered some undocumented behavior. I migrated on my store and found several issues.

The key point: Horizon themes will affect every store. Horizon themes deprecated without warning.

1. Here is the thing: AI merchandising is changing your app subscriptions.
2. The reality is that Shopify does not surface this clearly.
3. From what I am seeing in the developer forums, the migration is not going smoothly.

Bottom line: The situation is that most merchants do not know this is happening.