A/B Testing Done Right: Why Most Experiments Never Move the Needle

A/B testing has a reputation problem. Teams run dozens of experiments, declare a string of victories, and then look up to find revenue has barely moved. The tool isn’t broken. The way most people use it is.

Start with a reality check that reframes everything: only about one in six A/B tests produces a statistically significant winner — roughly 17%. Around a fifth of tests produce a clear winner overall; the rest are flat, inconclusive, or actively worse. If that sounds discouraging, it shouldn’t. It’s the whole point. Testing isn’t a machine that manufactures lifts on demand; it’s a filter that tells you which of your ideas actually work before you bet the site on them. Most ideas don’t. Finding that out cheaply, with a test, is the win.

Modest wins, compounded

The other myth worth killing is the dramatic case study. The median winning A/B test lifts conversion by about 6%, with a mean closer to 8% — not the “+217%!” headlines that litter the industry. That can feel underwhelming until you do the compounding. A series of 5–10% wins, stacked across a year, is how serious optimisation programmes pull away from competitors. The businesses that win at CRO aren’t the ones chasing one heroic redesign; they’re the ones banking small, verified gains, month after month.

Stop peeking

The most common and most expensive mistake is checking results too early. Nearly half of teams — 47% — look at their tests daily and stop the moment they see a “winner.” Doing that inflates false positives by three to five times. You see an early lift, call it, ship it — and the “win” evaporates in production because it was noise, not signal. The discipline is dull but decisive: decide your sample size before you start, and don’t touch the test until you reach it.

What does “enough” look like? As a working guide, aim for around 30,000 visitors and 3,000 conversions per variant, and let a test run a minimum of two weeks — to cover at least one full weekly cycle — but no longer than six to eight. Hold to 95% statistical significance and 80% power. Those aren’t academic niceties; they’re the difference between a result you can bank and a coin flip you mistook for insight.

Do you even have the traffic?

Here’s the honest caveat the testing-tool vendors gloss over: A/B testing only works if you have the traffic to reach significance in a reasonable time. A page with a few hundred visitors a week will take months to call a test, by which point the result is stale. If that’s you, A/B testing is the wrong first tool. Lean on qualitative methods instead — session recordings, user interviews, heuristic review — to find obvious friction, fix it, and save formal testing for when you have the volume to use it properly. Knowing when not to test is part of doing it right.

Test things that matter

Finally, point your limited testing capacity at high-impact questions, not button colours. The experiments worth running are the ones tied to a real hypothesis about why people hesitate — a confusing checkout step, a weak value proposition, a pricing page that buries the plan most people want. Prioritise by potential impact and the strength of your evidence that something is wrong. A handful of well-aimed tests on the pages that matter will beat a hundred trivial ones every time.

It’s worth remembering how rare this discipline still is. Only around 0.2% of websites run structured experiments at all. Done properly — patiently, with real statistical rigour, aimed at the things that move money — testing isn’t just a tactic. It’s a durable advantage over the overwhelming majority of competitors who are still guessing.