A proper two-tailed z-test on two proportions. Enter both variants and find out whether the difference is a result or a coin flip you got excited about.
This runs a two-proportion two-tailed z-test. It pools both variants to estimate a shared conversion rate, calculates the standard error of the difference, and converts the resulting z-score into a p-value using the standard normal distribution.
Confidence is simply one minus that p-value. At 95% confidence, there is roughly a one in twenty chance you would see a difference this large if the two variants were genuinely identical.
What significance does not mean. It does not mean B is better. It means the difference is unlikely to be pure chance. A statistically significant 0.2% lift on a page nobody important visits is still not worth shipping, and a non-significant result is not proof the change failed — usually it means you have not collected enough data yet.
Someone runs a test for four days, sees 12 conversions against 8, calls a 50% lift and ships it. Put those numbers in above and watch the confidence figure. That gap is comfortably inside what random variation produces, and shipping on it means shipping noise.
The opposite error is just as expensive: killing a genuinely good variant at day three because it was briefly behind. Both come from reading a difference before there is enough data to read.
This is the part most people skip, and it decides whether a test can work at all. Smaller effects need dramatically more traffic to detect, and the relationship is not linear.
| Baseline rate | Lift you want to detect | Visitors per variant |
|---|---|---|
| 3% | +50% relative — a big, obvious change | ~2,500 |
| 3% | +20% relative — a solid win | ~13,900 |
| 3% | +10% relative — a modest improvement | ~53,200 |
| 2% | +20% relative | ~21,100 |
| 1% | +30% relative | ~19,800 |
Calculated at 95% significance and 80% statistical power, which are the conventional defaults. Double the per-variant figure for the total traffic your test needs.
The uncomfortable implication: if you have 3,000 visitors a month, you cannot detect a 10% improvement in any reasonable timeframe. That is not a reason to give up. It is a reason to test bigger changes, or to make obviously-correct fixes without testing them at all — a broken mobile form does not need a control group.
Most businesses are not. There is a great deal you can fix without a control group, and we will tell you honestly which category you fall into.
Get a free conversion reviewAll free, all in-browser, none of them ask for an email.
What a one-point improvement is worth, per year, in money.
Open the calculator →A/B Test SimulatorA real significance test. Find out whether your winner is a winner.
You are hereForm Friction CalculatorCompound drop-off per field. What each extra question costs you.
Check your form →Every service page carries its own tools too — a reach planner on CTV, a deliverability checker on email, an incrementality model on retargeting and a real-time bidding visualizer on programmatic.
It runs a two-proportion two-tailed z-test. Both variants are pooled to estimate a shared conversion rate, the standard error of the difference is calculated, and the resulting z-score is converted to a p-value against the standard normal distribution. Confidence is one minus that p-value. The error function uses the Abramowitz and Stegun approximation, accurate to roughly one part in ten million.
95% is the conventional default and what this tool marks as significant. It means roughly a one in twenty chance of seeing a difference this large if the variants were genuinely identical. Some teams use 90% for low-risk changes and 99% for anything expensive to reverse. What matters more than the threshold is deciding it before the test rather than after seeing the data.
Far more than most people expect, and it depends on your baseline rate and the size of the effect you want to detect. At a 3% baseline, detecting a 50% relative lift needs around 2,500 visitors per variant; a 20% lift needs about 13,900; a 10% lift needs roughly 53,200. If you have a few thousand visitors a month you cannot detect small improvements in any reasonable timeframe, which is a reason to test bigger changes rather than to give up.
No, and this is the most common way teams manufacture false positives. Checking repeatedly and stopping the moment a test crosses 95% substantially inflates the chance of a spurious result, because with enough peeks almost any test crosses the line temporarily. Decide the sample size before starting and run to it.
Usually it means there is not enough data yet rather than that the change failed. Check the required sample size for your baseline and target lift. It can also mean the change was genuinely too small to matter, which is useful information — it tells you to test bolder changes rather than tweaking button colors.
