Why Essential A/B Test Traffic Isn’t Exactly 50/50 (or other chosen Traffic split), and why not all visitors are counted

If you’ve activated an A/B test in Essential A/B Testing and noticed that traffic is not split evenly with a 50/50 (or other chosen Traffic split) setup, don’t worry, this behavior is expected and explained by how tracking, cookie consent, and traffic assignment work.

Below we’ll walk you through everything step by step 👇

📌 Important Things to Know Before You Start

• Essential A/B Testing only tracks visitors who accept the cookie consent banner
• Only visitors who actually interact with the test template are included
• A 50/50 split means probability-based assignment, not a guaranteed equal count
• Small or medium traffic volumes can naturally show noticeable variation

1️⃣ Why Traffic Is Not Exactly 50/50 Between Variants

When you select 50% A / 50% B, this does not mean that:

• Exactly half of visitors will be assigned to Variant A

• Exactly half of visitors will be assigned to Variant B

  
  

Instead, it means:

• Each visitor has a 50% probability of being assigned to Variant B every time someone enters your website

• Assignments are made independently, without balancing against previous visitors

  
  

Because of this:

• Variation is normal and expected

• Especially with limited traffic, splits like 60/40 or 65/35 can happen naturally

  
  

Over time and with higher traffic, the distribution typically becomes more balanced.

2️⃣ How Cookie Consent Affects Traffic Split and why not all visitors are counted in the A/B Tests

Cookie consent plays a critical role.

Important points:

• We only receive data from Shopify after a visitor accepts cookies
• Visitors who never accept cookies are not tracked

• If one theme or variant makes cookie acceptance less visible or less necessary, it may result in fewer tracked visitors for that variant

  
  

This can affect:

• Total visitor counts
• Apparent imbalance between Variant A and B
• Number of recorded orders

3️⃣ Why This Is More Noticeable Right After Activating a Test

Right after activating a test:

• Traffic volume is usually still low
• Small differences feel more significant

• Cookie acceptance rates may not yet be consistent

  
  

Because of this, it’s normal to see:

• Fewer counted orders initially

• Uneven traffic distribution early on

  
  

We recommend letting the test run longer before drawing conclusions.

✅ Final Summary

If your A/B test shows an uneven 50/50 traffic split

This is usually because:

• Only visitors who accept cookies are tracked
• Only visitors who interact with the tested template are included
• 50/50 split is probability-based, not guaranteed

• Early-stage tests naturally show more variation

  
  

This behavior is expected and does not indicate a malfunction of the app.

If you still believe something looks incorrect after letting the test run longer, our support team is always happy to investigate further 💙