The paradox of A/B testing is that the teams who run the most tests are often the ones most confidently wrong. Because once you have a process, it's easy to trust the output without questioning whether the process itself is sound.
Here are the mistakes that quietly invalidate the data, and how to fix them before you make another major product decision on a false positive.
Peeking
This is the one everyone knows about and almost everyone still does. You launch a test on Monday, check it Wednesday because you're excited, and by Thursday you're wondering whether to call it early because variant B is up 12%.
The problem: every time you check an in-progress test and consider stopping it, you're performing an unplanned interim analysis. Repeated peeking changes the chance of a false positive, so a result that looks convincing early may be noise rather than a durable variant effect.
The fix is boring but works: calculate your required sample size before you start, commit to it, and don't look at the results until you've hit it. If that feels impossibly slow, the real issue is that your traffic volumes aren't sufficient for the pace of testing you're trying to run.
Statistical method: How not to run an A/B test
Too many variants
Testing many headline variants simultaneously feels efficient. It's often the opposite. Each additional variant dilutes your traffic and increases the probability that one of them looks like a winner by chance. Predefine the comparison and adjust the analysis for the number of variants.
The constraint forces better thinking anyway. If you have five headline ideas, ranking them by expected impact and testing the top one means you've already done the analysis. A good prioritisation framework beats a wide test every time.
One hypothesis. One variant. One primary metric. That's the test. Everything else is a research project dressed up as an experiment.
Ignoring segment effects
A test that shows a flat overall result might be hiding something significant underneath. A headline change that converts desktop users 18% better might be actively hurting mobile, and the two effects cancel each other out in aggregate.
Segment your results by device type before calling any result. This alone would change the conclusion of a meaningful number of the tests run by most teams. Source, new vs. returning, and geography are the next cuts worth making. The aggregate result is the starting point, not the answer.
Running tests during anomalous periods
A test that runs over a bank holiday, a product launch, a viral moment on social, or any other traffic anomaly is capturing a non-representative sample. Your results reflect those visitors, not your typical audience.
This sounds obvious until it happens during a test you really care about. The temptation to include the anomalous data is strong, especially when the sample size is finally where you want it. Exclude it anyway. The extra week of runtime is cheaper than a bad decision at scale.
Changing the test after it starts
Mid-test edits - whether it's fixing a bug in the variant, adjusting targeting, or tweaking the copy because someone in leadership didn't like the original - split your test into two different experiments with shared reporting. The data before and after the change can't be combined without invalidating both.
If the change is essential, restart. Log the reason, reset your sample size calculation, and treat the pre-change data as a separate analysis. It's frustrating. It's also the only way to have data you can trust.
What rigorous testing actually looks like
- Sample size calculated in advance based on your baseline CVR and minimum detectable effect
- Runtime committed to before launch, regardless of what the numbers look like mid-test
- Results segmented by device, source, and visitor type before any decision is made
- A clear primary metric defined before the test starts (secondary metrics are informative, not decisive)
- Documented learnings, including losses - a losing test that teaches you something is more valuable than a win you don't understand
Most of the above is process, not technology. The hardest part is having the discipline to follow it when the data looks compelling early. That discipline is what separates teams that actually improve conversion rates from teams that just run a lot of tests.