One of the most common questions when planning an A/B test is: how many users do I need to get a reliable result? The answer is not a magic number: it depends on the size of the effect we want to detect, the baseline conversion rate, and the level of statistical certainty we require.
Calculating the sample size in advance is essential to avoid two classic mistakes: stopping the test too early and declaring a winner that does not exist, or letting it run too long, wasting traffic and time. In other words, it is about finding the right balance between resources and rigour.
If you have read the article on A/B Testing, you will recall that power analysis is the statistical method that lets us determine this threshold. And if you have studied confidence intervals, you already know that significance level and test power are not abstract concepts but operational levers that directly affect sample size.
The calculator below automates this process: simply enter your test parameters to instantly get the number of observations needed per variant and, if you know your daily traffic, an estimate of the test duration in days.
What We’ll Cover
The calculator
Enter the parameters of your A/B test and the calculator will instantly return the required sample size.
Sample Size Calculator
The current conversion rate of the control variant
The smallest relative improvement we consider meaningful (e.g. 20% = from 5% to 6%)
Total daily visitors to estimate test duration
—
The formula: how the calculation works
The calculator uses the standard formula for comparing two proportions with a two-tailed z-test. Let us walk through it step by step.
We start with the parameters we enter:
- p1: the baseline conversion rate (control), expressed as a proportion. If our CR is 5%, then p1 = 0.05.
- p2: the expected conversion rate for the variant. If the minimum detectable effect (MDE) is 20% relative, then p2 = p1 × (1 + MDE/100) = 0.05 × 1.20 = 0.06.
- α: the significance level, i.e. the probability of declaring an effect when there is none (Type I error). With α = 0.05 we work at 95% confidence.
- 1 − β: the power of the test, i.e. the probability of detecting an effect when it actually exists. With power 0.80, we have an 80% chance of catching the effect.
The formula is:
\( n = \frac{\left[z_{\alpha/2} + z_{\beta}\right]^2 \cdot \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 – p_2)^2} \)Where zα/2 and zβ are the quantiles of the standard normal distribution. For the most common values:
- α = 0.05 → zα/2 = 1.96
- α = 0.01 → zα/2 = 2.576
- β = 0.20 (power 0.80) → zβ = 0.842
- β = 0.10 (power 0.90) → zβ = 1.282
Worked example. Suppose we have a baseline conversion rate of 3% and we want to detect a 20% relative increase (i.e. going from 3% to 3.6%), with α = 0.05 and power = 0.80:
- p1 = 0.03, p2 = 0.036
- zα/2 = 1.96, zβ = 0.842
- Numerator: (1.96 + 0.842)2 × [0.03 × 0.97 + 0.036 × 0.964] = 7.849 × 0.0638 = 0.5008
- Denominator: (0.03 − 0.036)2 = 0.000036
- n = 0.5008 / 0.000036 ≈ 13,911 per variant
So to detect a 20% relative effect on a 3% CR, we need roughly 13,900 observations per variant (nearly 28,000 in total). These numbers are worth reflecting on: if our site gets 500 visitors a day, the test will take about 56 days. This is one of the reasons why, in practice, most A/B tests on medium-traffic sites take weeks, not days.
How to use the calculator
How to choose the MDE. The minimum detectable effect is the trickiest parameter. Rather than asking “how much would we like the metric to improve”, we should ask: what is the smallest improvement that would justify the effort of implementing the change? An MDE of 5% relative requires enormous samples; an MDE of 50% is easy to detect but rarely realistic. The 10–30% range is a good starting point for most conversion rate tests.
An important detail: the MDE in the calculator is relative, not absolute. An MDE of 20% on a baseline CR of 5% means we are looking to detect a shift from 5% to 6% (one absolute percentage point, but 20% of the starting value).
How to estimate daily traffic. The traffic to enter is that of the pages involved in the test, not the total site traffic. If the test is on the checkout page and it receives 300 visits per day, the correct value is 300. You can get this figure from your analytics tool (GA4, Matomo, or similar) by averaging the last 30 days to smooth out daily fluctuations.
You might also like
Further reading
The most comprehensive reference on the rigorous design of online experiments is: Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu. It covers sample size, power analysis and much more, drawing on decades of practical experience at Microsoft and Google.