Interpreting Your Test Results
Once your A/B test is live, the app continuously monitors visitor behavior and updates results in real time. This guide explains how to understand what you're seeing and how to know when it's time to act.
What Is “Win Chance”?
We use a Bayesian statistical model to calculate test results. This approach gives you a continuously updating Win Chance, which tells you the probability that your variant is actually better than the original.
What Win Chance means:
- Win Chance = 87% → There’s an 87% probability that your variant is better than the original.
- Win Chance = 20% → There’s an 80% probability that the original is better.
The model always calculates from the perspective of the variant.
How a winner is declared:
Once any variant reaches a 95% Win Chance, the test is marked Statistically significant, and that variant is highlighted as the winner.
You’re not required to wait until 95%. The Win Chance gives you flexible guidance — you can choose to act earlier depending on the uplift, business goals, and traffic availability.
Recommended actions based on Win Chance
| Win Chance Range | What it means | What to do |
|---|---|---|
| < 50% | Variant is underperforming | Consider stopping the test early and sticking with the original. |
| 50–80% | Inconclusive / directional | Let the test run. The model needs more data. |
| 80–95% | Promising “leaning winner” | Decide if the uplift justifies ending early or waiting for full significance. |
| ≥ 95% | Statistically significant | Confident winner — safe to publish the variant. |
Test Status
Every test has two status types: test phase and statistical status.
Test phase:
- Draft – You've created a test but haven’t started it yet.
- Live – Test is active and collecting visitor data.
Statistical status:
- Collecting data – The test has not yet collected enough data to determine whether the result is significant or unlikely.
- Statistically significant – A variant has reached a 95% Win Chance. The system considers it a likely long-term winner.
- Significance unlikely – The test has been running for over 2 weeks, and the model projects it would take more than 60 additional days to reach 95% Win Chance. The result is too uncertain — consider ending the test and trying a new variant.
Main Statistics (Top Section)
These are high-level metrics visible at the top of your test results page:
- Total visitors – Combined number of visitors exposed to both A - original, and B - version of your test.
- Test duration – How much time the test has been running.
- Improvement – The difference in performance between the variant and the original, based on your selected goal (e.g., conversion rate).
- Win Chance – The current probability that the variant is better than the original, based on the data so far.
Test Progress
This section helps you understand how far along the test is toward significance:
- Progress bar – Visually tracks progress toward either:
- The minimum required threshold (3 days or 30 conversions/variant)
- The statistical significance threshold (95% Win Chance)
- Estimated X more orders needed – A model-based forecast of how many more conversions are likely needed to reach 95%.
- ~N days left – Based on current data volume, how many days are likely needed to reach statistical significance.
Analytics Breakdown
This section shows detailed metrics for each variant side-by-side:
| Metric | Description |
|---|---|
| Variant Win Chance | Live probability (Bayesian) that the variant is outperforming the original. |
| Visitors | Number of unique users who saw that version of the template. |
| Add to Carts | The number of visitors who added items to their carts |
| ATC Rate | Add to Cart events divided by number of visitors – shows how effectively each version gets users to start the purchase flow. |
| Orders | Number of completed orders tracked for each variant. |
| Conversion Rate | Percentage of converted visitors – a key metric for many tests. |
| Revenue | Total sales amount generated by each version. |
| Revenue per Visitor (RPV) | Revenue ÷ Visitors – useful for evaluating true revenue performance beyond just conversion rates. |
| Average Order Value (AOV) | Revenue ÷ Orders – shows whether one version leads to bigger baskets or more premium purchases. |
Each of these metrics is visualized in bar charts below the analytics table, with up/down arrows and color coding to help you quickly spot trends.
Final Notes
If you’re not sure how to act on the results:
- A strong Win Chance + high uplift = publish the winner.
- Low Win Chance or flat improvement = iterate and try a new variation.
- If your Win Chance is hovering for many days, you might be seeing small/no differences — which is still useful insight!
Still have questions? Contact our support team — we're happy to help you.