A/B test statistical significance calculator

Enter visitors and conversions for each version to see whether the difference is real. Planning a test? Switch to sample size to see how much traffic you need.

Version A (control)

Version B (variant)

Confidence
98.2%
Significant at 95%. Version B beat version A, and a gap this size is unlikely to be chance.
Conversion rate A
3.00%
95% range 2.68% to 3.35%
Conversion rate B
3.60%
95% range 3.25% to 3.98%
Relative lift
20.0%
p-value (two-sided)
0.0175

How statistical significance is calculated

The calculator compares the two conversion rates with a two-proportion z-test. It measures the gap between them in standard errors:

p = (conversions A + conversions B) ÷ (visitors A + visitors B)
SE = √( p × (1 − p) × (1 ÷ visitors A + 1 ÷ visitors B) )
z = (rate B − rate A) ÷ SE

The p-value is the chance of a z this far from zero, in either direction, if the versions were really the same. Confidence is 1 minus the p-value.

Worked example

Version A converts 300 of 10,000 visitors (3.0%). Version B converts 360 of 10,000 (3.6%), a 20% relative lift.

z = 2.38, p-value = 0.018, confidence = 98.2%

That clears the 95% bar, so the lift is unlikely to be noise. With 1,000 visitors per version and the same rates (30 against 36), the result would be nowhere near significant: same lift, a tenth of the evidence.

How to calculate A/B test sample size

n per version = (zα/2 + zβ)² × (p₁(1 − p₁) + p₂(1 − p₂)) ÷ (p₂ − p₁)²

At 95% significance and 80% power, zα/2 is 1.96 and zβ is 0.84. To detect a 10% relative lift on a 3% conversion rate (3.0% → 3.3%), you need 53,208 visitors per version, 106,416 in total. Divide by daily traffic to estimate how long the test has to run.

With Ordinary

Significance on orders, revenue and profit, not just clicks.

A conversion-rate test is only half the answer. A variant that converts more often at a lower basket size can still bring in less money.

Ordinary runs storefront A/B tests on Shopify and reads them against real orders: conversion rate and revenue per visitor for each version, with a confidence range that shows how sure the result is.

Connect your store How A/B testing works

Frequently asked questions

What does statistically significant mean in an A/B test?

It means the difference between the versions is large enough, for the number of visitors, that it's unlikely to be random chance. At 95% significance, a gap this size would appear less than 5% of the time if the two versions truly converted at the same rate.

What is a p-value?

The p-value is the probability of seeing a difference at least this large if there were no real difference between the versions. A p-value below 0.05 is significant at 95%. It is not the probability that version B is better.

Why does this calculator use a two-sided test?

A two-sided test counts a surprising result in either direction, so a variant that loses badly is flagged as clearly as one that wins. One-sided tests reach significance sooner but double the false-positive rate for anyone who looks at the data before choosing a direction.

Can I stop a test as soon as it reaches significance?

Not safely. Checking repeatedly and stopping at the first significant reading inflates false positives well above 5%. Decide the sample size before you start, using the sample size tab, and read the result when you reach it.

How many visitors do I need for an A/B test?

It depends on your current conversion rate and the smallest lift you care about. Detecting a 10% relative lift on a 3% conversion rate takes about 53,000 visitors per version at 95% significance and 80% power. Halving the lift roughly quadruples the traffic needed.

What test does this calculator use?

A pooled two-proportion z-test on conversion rates, two-sided, with Wilson score intervals for each version's conversion rate. Sample size uses the standard two-proportion power formula with a 50/50 split.

More free calculators