> Source: https://tryordinary.com/tools/statistical-significance-calculator [Free calculators](/tools) # A/B test statistical significance calculator Enter visitors and conversions for each version to see whether the difference is real. Planning a test? Switch to sample size to see how much traffic you need. Check a resultSample sizeVersion A (control) VisitorsConversionsVersion B (variant) VisitorsConversionsConfidence98.2%Significant at 95%. Version B beat version A, and a gap this size is unlikely to be chance.Conversion rate A3.00%95% range 2.68% to 3.35%Conversion rate B3.60%95% range 3.25% to 3.98%Relative lift20.0%p-value (two-sided)0.0175 ## How statistical significance is calculated The calculator compares the two conversion rates with a two-proportion z-test. It measures the gap between them in standard errors: p = (conversions A + conversions B) ÷ (visitors A + visitors B) SE = √( p × (1 − p) × (1 ÷ visitors A + 1 ÷ visitors B) ) z = (rate B − rate A) ÷ SE The p-value is the chance of a z this far from zero, in either direction, if the versions were really the same. Confidence is 1 minus the p-value. ### Worked example Version A converts **300** of **10,000** visitors (3.0%). Version B converts **360** of **10,000** (3.6%), a 20% relative lift. z = 2.38, p-value = 0.018, confidence = 98.2% That clears the 95% bar, so the lift is unlikely to be noise. With 1,000 visitors per version and the same rates (30 against 36), the result would be nowhere near significant: same lift, a tenth of the evidence. ## How to calculate A/B test sample size n per version = (zα/2 + zβ)² × (p₁(1 − p₁) + p₂(1 − p₂)) ÷ (p₂ − p₁)² At 95% significance and 80% power, zα/2 is 1.96 and zβ is 0.84. To detect a **10%** relative lift on a **3%** conversion rate (3.0% → 3.3%), you need **53,208** visitors per version, 106,416 in total. Divide by daily traffic to estimate how long the test has to run. With Ordinary ## Significance on orders, revenue and profit, not just clicks. A conversion-rate test is only half the answer. A variant that converts more often at a lower basket size can still bring in less money. Ordinary runs storefront A/B tests on Shopify and reads them against real orders: conversion rate and revenue per visitor for each version, with a confidence range that shows how sure the result is. [Connect your store](https://apps.shopify.com/ordinary-attribution-ab-testing) [How A/B testing works](/solutions/ab-testing) ## Frequently asked questions ### What does statistically significant mean in an A/B test? It means the difference between the versions is large enough, for the number of visitors, that it's unlikely to be random chance. At 95% significance, a gap this size would appear less than 5% of the time if the two versions truly converted at the same rate. ### What is a p-value? The p-value is the probability of seeing a difference at least this large if there were no real difference between the versions. A p-value below 0.05 is significant at 95%. It is not the probability that version B is better. ### Why does this calculator use a two-sided test? A two-sided test counts a surprising result in either direction, so a variant that loses badly is flagged as clearly as one that wins. One-sided tests reach significance sooner but double the false-positive rate for anyone who looks at the data before choosing a direction. ### Can I stop a test as soon as it reaches significance? Not safely. Checking repeatedly and stopping at the first significant reading inflates false positives well above 5%. Decide the sample size before you start, using the sample size tab, and read the result when you reach it. ### How many visitors do I need for an A/B test? It depends on your current conversion rate and the smallest lift you care about. Detecting a 10% relative lift on a 3% conversion rate takes about 53,000 visitors per version at 95% significance and 80% power. Halving the lift roughly quadruples the traffic needed. ### What test does this calculator use? A pooled two-proportion z-test on conversion rates, two-sided, with Wilson score intervals for each version's conversion rate. Sample size uses the standard two-proportion power formula with a 50/50 split. ## More free calculators - [01 / Calculator↗ ROAS calculator Return on ad spend from revenue and spend, in one step.](/tools/roas-calculator) - [02 / Calculator↗ Break-even ROAS calculator The lowest ROAS that doesn't lose money, from your real costs.](/tools/break-even-roas-calculator) - [03 / Calculator↗ CAC calculator What it costs to win one new customer, paid and fully loaded.](/tools/cac-calculator) - [04 / Calculator↗ Customer lifetime value calculator LTV in revenue and gross profit, plus your LTV:CAC ratio.](/tools/customer-lifetime-value-calculator) - [05 / Calculator↗ Profit margin calculator Margin and markup from price and cost, or the price for a target margin.](/tools/profit-margin-calculator)