How it works
Traffic is split at random into two groups. Group A sees the current version: that's the control group. Group B sees the new version. Because the split is random, the only difference between the groups is the change being tested.
For a test to be reliable:
- The metric is chosen before launch: conversion rate, revenue per visitor, average order value.
- The test runs long enough to cover full weeks, because people don't shop the same way on a Tuesday and a Sunday.
- The gap you see is statistically significant, not random noise.
- You change one thing at a time.
For an AI Shopping Assistant, group B gets the assistant and group A doesn't. The revenue gap between the two groups is the incremental revenue. The assistant only runs on sessions that accepted cookies, so the test and the measured gain apply to that share of traffic.
Why it matters in e-commerce
Before/after comparisons are misleading in e-commerce: sales events, campaigns, weather and seasonality move revenue from one week to the next. An A/B test cancels these effects out, since both groups live through the same period. That's what lets you say a gain really comes from the change, and put a solid number on ROI.
Real-life example: Samsonite
Samsonite measured its AI Shopping Assistant with an A/B test. The result: a 2.2x incremental ROI, meaning the incremental revenue the assistant brought in was 2.2 times its total cost.
Common confusion
Comparing shoppers who used the assistant with those who didn't is not an A/B test. The two groups aren't random: shoppers who ask a question are often further along in their purchase. That comparison gives you the AI-assisted conversion rate, which is useful for steering, but it doesn't measure incremental impact.