Skip to content

A/B testing

E-commerce metrics

A/B testing compares two versions of a page or experience by splitting shoppers at random between them, to measure which one performs better. It's the standard method for isolating the real effect of a change, such as adding an AI Shopping Assistant.

How it works

Traffic is split at random into two groups. Group A sees the current version: that's the control group. Group B sees the new version. Because the split is random, the only difference between the groups is the change being tested.

For a test to be reliable:

  • The metric is chosen before launch: conversion rate, revenue per visitor, average order value.
  • The test runs long enough to cover full weeks, because people don't shop the same way on a Tuesday and a Sunday.
  • The gap you see is statistically significant, not random noise.
  • You change one thing at a time.

For an AI Shopping Assistant, group B gets the assistant and group A doesn't. The revenue gap between the two groups is the incremental revenue. The assistant only runs on sessions that accepted cookies, so the test and the measured gain apply to that share of traffic.

Why it matters in e-commerce

Before/after comparisons are misleading in e-commerce: sales events, campaigns, weather and seasonality move revenue from one week to the next. An A/B test cancels these effects out, since both groups live through the same period. That's what lets you say a gain really comes from the change, and put a solid number on ROI.

Real-life example: Samsonite

Samsonite measured its AI Shopping Assistant with an A/B test. The result: a 2.2x incremental ROI, meaning the incremental revenue the assistant brought in was 2.2 times its total cost.

 

  • +15%

    incremental revenue on AI-guided shopper experiences

  • 2.2x

    incremental ROI at Samsonite, proven by A/B test

Common confusion

Comparing shoppers who used the assistant with those who didn't is not an A/B test. The two groups aren't random: shoppers who ask a question are often further along in their purchase. That comparison gives you the AI-assisted conversion rate, which is useful for steering, but it doesn't measure incremental impact.

The iAdvize approach

Gains measured against a control group

The +15% incremental revenue iAdvize reports is measured with A/B tests. The ROI calculator gives you an estimate for your own site, and reminds you that the assistant only runs on sessions that accepted cookies.

 

Frequently asked questions

How long should an A/B test run?

Long enough to reach a statistically significant result, and at least full weeks. Duration depends mostly on traffic: the lower it is, the longer the test.

Which metric should you track when testing an AI Shopping Assistant?

Revenue per visitor, because it combines conversion rate and average order value. Tracking conversion alone can hide a drop in basket size.

Does an AI Shopping Assistant replace A/B testing on the site?

No, it works alongside it. Teams keep testing their pages, and the assistant itself can be evaluated with an A/B test.