Reading your results

Open an experiment from Experiments to see its Results panel. It shows one allocation version at a time (see below) and is read when you open the page; select Refresh for the latest figures.

Visitors and clicks

The first table counts traffic per variant:

• Visitors — distinct people who clicked the smart link and were sent to the variant. A visitor is recognised by a cookie the link sets, so the same person clicking twice is one visitor.
• Clicks — every click, repeat visits included.
• Share — the variant’s share of the visitors assigned in this allocation version.
• Weight — the share you configured. Share and weight should be close; see SRM warning when they are not.

Some clicks are left out of every figure, and the line under the table says how many:

• bot — clicks from known crawlers and bots;
• fallback-served — visitors assigned to a variant whose destination couldn’t be served, who got the fallback URL instead. They didn’t see the variant, so counting them would blur the comparison;
• while paused — clicks while the experiment was paused, which all went to the control.

Orders and revenue

Once your Shopify store is connected, the second table shows what the visitors bought, attributed under rule v1:

• Orders — attributed orders of the variant’s visitors. Refunded orders count, at their net value; cancelled orders don’t.
• Conversion — the share of visitors who placed at least one order, with its 95% credible interval underneath. A visitor who orders twice converts once.
• AOV — average order value: revenue divided by orders.
• Revenue — the attributed orders’ subtotal, net of refunds.
• RPV — revenue per visitor: revenue divided by visitors, including everyone who bought nothing.
• vs control — how the variant compares with the control: the difference in RPV with its 95% interval underneath, and how likely its conversion is higher or lower than the control’s.

All amounts are in your shop currency. If orders arrive in more than one currency, the figures use the currency with the most orders, and the note under the table names the orders left out; amounts in different currencies are never added up. Shopify test orders are left out, with a note saying how many.

Why RPV is the main number

A page can raise conversion by selling cheaper baskets, or raise order value while fewer people buy. RPV combines both: it is what an average visitor is worth. The readouts and the traffic estimate are about RPV.

Probability and intervals

Under the table there is one sentence for each variant other than the control, for example:

Variant B is ahead of Control on RPV, 78% probability; not yet conclusive.

It means: given the visitors and orders so far, there is a 78% chance that Variant B’s true RPV is higher than the control’s. It is a statement about the evidence, not a verdict:

• 78% is not a sure thing. One time in five or so, a variant this far ahead is not really better.
• “not yet conclusive” stays until both the 95% interval of the RPV difference excludes zero and the experiment has enough traffic (see below). Then the sentence ends with “conclusive at this traffic”.
• “level” means neither side is ahead by more than a coin flip yet.
• “Not enough data yet” appears while the variant or the control has no orders (every visitor spent nothing, so there is nothing to compare) or fewer than two visitors.

The intervals are ranges the true value is likely to fall in (95%). Wide intervals mean little data; they narrow as visitors and orders accumulate.

GrowthFlow never picks a winner or changes your split for you. When to stop, and what to keep, is your decision; the panel gives you the evidence for it.

Traffic sufficiency

While the experiment runs, a line estimates how much longer it needs, for example:

Detecting a 10% change in RPV needs about 12 more days at the current traffic (8,400 visitors per variant).

It sizes the experiment to detect a 10% difference in RPV reliably, from how much your visitors’ spending varies today and how fast the smallest variant gets visitors. “Enough traffic” means every variant has reached that size. Until there are orders, it says it needs orders first; until every variant has visitors, it needs visitors on every variant first.

Stopping before then is the most common way to fool yourself: early leads are often noise, and a result that looks decisive after three days frequently evaporates by the second week.

SRM warning

The sample ratio mismatch check compares how visitors were actually split with the weights you configured. A small difference is normal; a difference that chance can’t explain is flagged:

Sample ratio mismatch: the split visitors got (62/38) differs from the configured 50/50 more than chance explains (p < 0.001). Don’t trust this segment’s results until the cause is found.

When the split is fine, it says Split as configured.

Visitors are counted the moment the smart link assigns them, before any page loads, and bots are left out. So a slow, broken or redirecting destination page can’t cause a mismatch, and neither can people visiting a variant’s page directly (they never pass through the smart link). A mismatch is rare: about one check in a thousand flags by chance, and otherwise it points to a fault in how the split was recorded. Note when it started, don’t act on that segment’s results, and ask your account contact.

The check waits until every variant expects at least 5 visitors.

Allocation versions

The weights of a running experiment can’t change silently. When you change them, the app asks Reassign every visitor?; confirming with Reassign and save starts a new allocation version (v2, v3, …): every visitor is assigned again under the new weights, and the results start over for that version. Results of different versions are never mixed, because visitors were split differently.

The Allocation selector on the Results panel shows any earlier version with the weights it ran with. The traffic estimate is shown for the current version only.

Pause, resume, end and rollback

• Pause sends every visitor to the control; those clicks are counted as “while paused” and left out of the comparison. Resume continues the same allocation version.
• End experiment stops the experiment for good; visitors go to the link’s own destination (or fallback URL). The results stay readable.
• Roll back to v… (on the domain’s Config history, opened with History on Domains) puts an earlier published configuration back on the edge in an emergency; the next save or Publish now puts your saved settings back. Results are always counted by the allocation version each click was assigned under, so a rollback never mixes segments.

Troubleshooting

• The figures didn’t change. Results are read when the page opens; select Refresh.
• Orders are missing. Check that the store is connected on Integrations and see How attribution works for why an order may be unattributed.
• “Not enough data yet” for a long time. Both the variant and the control need at least one order before they can be compared.
• An SRM warning appeared. Don’t act on that segment’s results, note when the warning started, and ask your account contact; destination pages can’t cause it.
• The results reset. The weights changed, which started a new allocation version; pick the earlier one in the Allocation selector.