A/B testing offers

Compare two versions of an offer, read the results, pick the winner.

6 min read

Starting a test

Open a live offer (not a fixed bundle) and press A/B test, or press A/B in the Offers list.
MixMaxer duplicates it as “<name> (B)”, a draft in the same group. Change exactly one thing in B: the discount level, the tiers, the title or the layout.
Publish B. From now on visitors are split in half.

How the split works

The storefront assigns each visitor to A or B at random on the first view and remembers it in the browser, so a visitor always sees the same variant.
The choice is written to the cart, and the discount Function applies only the variant the visitor saw. POS, draft orders and visitors without a cart bucket get A.
Both variants rank as one offer in the priority order.
If one variant is paused, the other one shows to everyone; the split resumes when both are live.

Reading the results

Analytics › Experiments lists, per variant, impressions, orders, conversion, revenue, profit and the value per visitor, counted since the test started. Below the table a bar shows the probability that B beats A on the metric you chose. Pick A or Pick B keeps the winner live for everyone and pauses the loser; End without a winner detaches the two offers.

Choosing the metric

Conversion: orders per impression. Best when both variants sell the same products at similar prices.
Revenue per visitor: conversion × average order value. Use it when the variants differ in price or bundle size.
Profit per visitor: conversion × average profit per order. The honest metric when one variant gives a deeper discount: more orders can still mean less money. Needs “Cost per item” on your variants in Shopify (Products › variant › Cost per item).

Letting MixMaxer decide

Set the confidence (95% is the usual choice; 90% decides faster, 99% is stricter), the minimum orders per variant (30 by default) and tick “Apply the winner automatically”. When a new order pushes the probability past the confidence, the winner keeps running for everyone and the loser is paused, without anyone logging in. The probability comes from a Bayesian model of each variant's orders per impression, so it is valid at any moment: there is no fixed test length to wait for.

Holdout group: what offers really add

Settings › General › Holdout group (Pro) sends a share of visitors (for example 10%) past every offer: no widget, no cart bar, no badges, no automatic discount. Analytics › Holdout experiment compares the average order value of those visitors with everyone else's. Unlike the “bundle vs other orders” comparison, this is a true experiment: seasonality, campaigns and traffic mix hit both groups equally. Give it two to four weeks and 30+ orders per group, then switch the holdout off.

Common problems

Pick B is refused.

B must be published (Active) before it can win. Publish it or pick A.

One variant has far fewer impressions.

Normal in the first hours; visitors are assigned on their first product view, not per page load. Very uneven numbers over days mean one variant is hidden (paused, expired) or the product page caches per variant.

Can I test three variants?

No; two at a time. Run a second test against the winner afterwards.

Profit shows “—”.

No cost is set on the variants sold. Add “Cost per item” in Shopify (Products › variant › Pricing). Costs are read when an order arrives, so earlier orders stay without profit.

The automatic winner was applied too early / too late.

Raise the minimum orders or the confidence for a stricter decision; lower them to decide faster. The decision is re-checked with every order, never on a timer.

Holdout visitors still got a discount.

Discount codes and Shopify's own automatic discounts are not affected; only MixMaxer offers are withheld. Also, the holdout is per browser: the same person on another device may be in the other group.

More in Analytics & reports

← All articles · Next: Profit per offer and the holdout group · Still stuck? Contact support