MixMaxerEvery bundle app dashboard shows a big number: revenue generated, AOV uplift, orders influenced. The number is usually computed by counting every order that contains a bundle as "app revenue". Customers who would have bought two units anyway are counted. Orders from the ad campaign that launched on Tuesday are counted. The number is not wrong, it is just not an answer to the question you asked, which is: what would I have earned without the offers?
There is one method that answers that question, and it has been standard in paid marketing for a decade: a holdout.
What a holdout group is
A holdout is a small, random share of visitors who are shown no offers at all: no tiers, no bundle widget, no cart bar. Everyone else sees the offers as usual. Over the same days, with the same traffic and the same campaigns, you compare the average order value of the two groups. The difference is what the offers add. Nothing else differs, so nothing else can explain the gap.
Seasonality, a viral product, an email blast, a competitor's outage: all of these hit both groups equally. That is the point.
Why "this month vs last month" does not work
Suppose you turn on bundles on 1 October and AOV goes from $48 in September to $55 in October. Did bundles add $7?
- October had a new product launch.
- You raised ad spend on the 10th.
- Returning customers from a September sale came back.
- A shipping threshold changed.
You cannot separate the bundle effect from the rest. A before/after comparison is a story, not a measurement. A holdout is a measurement.
How big the holdout should be
Big enough to be read, small enough not to cost much.
| Monthly orders | Suggested holdout | Orders in holdout per month | Time to a readable result |
|---|---|---|---|
| 300 | 10% | 30 | 6–8 weeks |
| 1,000 | 10% | 100 | 3–4 weeks |
| 3,000 | 5% | 150 | 2–3 weeks |
| 10,000+ | 5% | 500+ | 1–2 weeks |
Rule of thumb: you want at least 100 holdout orders before you trust the number, and at least 200 before you make a pricing decision from it. Below 300 orders a month, a holdout will take a long time; a direct A/B test of one offer against a variant is more practical.
The cost is real but small. If your offers add $6 of AOV and 5% of visitors miss them, you forgo about 0.3% of revenue to learn whether the other 99.7% is working.
Setting one up on Shopify
Your bundle app has to support it; a holdout cannot be built with discount codes or theme settings. The app needs to:
- Assign each visitor to "offers" or "holdout" randomly and remember the assignment, so the same person never sees offers on one visit and not the next.
- Suppress every surface for the holdout: product widget, cart bar, badges, post-purchase offer. Half a holdout is no holdout.
- Still record the holdout group's orders and AOV, and show the comparison in analytics.
In MixMaxer this is a single setting on the Pro plan (Settings › Holdout share). Analytics then shows "AOV with offers", "AOV holdout" and the uplift, over the window you choose.
How to read the result
- Uplift is positive and stable across two consecutive windows: your offers work. Report that number, not the dashboard number.
- Uplift is positive in the first window and shrinks: early novelty, or a sale that overlapped. Keep the holdout running.
- Uplift is near zero: the offers are rewarding behaviour that already happened. Look at which tiers people take; if 90% of "buy 2" customers already bought 2 before, move the tier to 3.
- Uplift is negative: rare, but it happens when a widget slows the page or confuses the price. Check page speed and the "price on page vs checkout" question first.
Expect the honest number to be smaller than the dashboard number. That is normal. It is also the only number worth putting in a board deck.
Holdout vs A/B test: which one when
| Question | Tool |
|---|---|
| "Do my offers add anything at all?" | Holdout |
| "Is 10% or 15% on tier 2 more profitable?" | A/B test of that offer |
| "Should the free gift start at $80 or $100?" | A/B test |
| "Is the cart bar worth keeping?" | Holdout (it suppresses the bar too) |
Run both. The holdout tells you the size of the prize; A/B tests tell you how to grow it.
FAQ
What is a holdout group in e-commerce?
A random share of visitors who are deliberately not shown a promotion, used as a baseline. Comparing their behaviour with everyone else's over the same period measures the promotion's true effect.
How large should a holdout be for a Shopify store?
5–10% of visitors, sized so you collect at least 100 holdout orders before reading the result.
Does a holdout cost sales?
A little: the holdout group misses the uplift. At 5% holdout and a $6 AOV uplift the cost is roughly 0.3% of revenue.
Which Shopify bundle apps have a holdout group?
Very few. MixMaxer includes a holdout setting on its Pro plan; check your current app's analytics page for "holdout" or "control group" before assuming it has one.
MixMaxer: bundles, volume discounts and upsells for Shopify, with A/B tests, profit per offer and a holdout group. Install from the Shopify App Store · Profit and holdout help article.