Sampling and Sample Size: How Much Data Do You Really Need?

Two political polls: one on 800 people, one on 20,000. Is the second one obviously more reliable? Not necessarily. If the first was drawn randomly from the population and the second used self-selected online volunteers, the first can be far more accurate than the second.

Quality comes before quantity. It is a sentence worth remembering, because it overturns common intuition.

Two distinct questions

When planning a data collection, we need to answer two separate questions, and the first always comes before the second:

1. Does the sample truly represent the population? This depends on the sampling method, not on the amount of data.
2. Is the sample large enough? This depends on the variability of the data and the precision we need.

Question	Depends on
Is the sample representative?	Sampling method
How many observations are needed?	Variability, expected effect, confidence level, margin of error

In web marketing we face both problems constantly. An A/B test run only on mobile users tells us little about desktop users (representativeness issue), even if we collect millions of visits. A test with 100 visitors per variant tells us nothing useful (sample size issue), even if the sampling is perfectly random.

How to Choose Who to Measure: Types of Sampling

Before figuring out how much data we need, we must understand how to collect it. The three main methods are:

• Simple random sampling: Every user has exactly the same probability of being chosen. It’s the gold standard, what we try to achieve when we randomize users in an A/B test.
• Stratified sampling: We divide users into groups (e.g., Mobile and Desktop traffic) and sample randomly within each group, preserving the original proportions. It ensures no important minority is ignored.
• Systematic sampling: We pick every k-th user (e.g., one in 10). Easy to implement, but insidious when data has hidden periodicity (imagine sampling every 7th user: if we only pick Mondays, the estimate will be biased from the start).

In the normal distribution we saw that sampling variability decreases as observations increase — but only if the sample is representative. A large but biased sample gives precise but wrong estimates, and it is far worse than a small but well-collected one.

Sample Size: The Math Behind the Estimation

Now let us assume we have solved the representativeness problem. The sample is well collected. How large does it need to be?

The intuition is simple: the smaller the effect we are looking for (or the more variable the data), the more data we need to distinguish it from background noise. If all individuals in the population were nearly identical, a few observations would suffice. The more heterogeneous the population, the larger the sample needs to be to describe it accurately.

Why population size barely matters

This is one of the most surprising results in statistics. Whether a site has 10,000 visitors per month or 10 million, the sample needed to estimate a proportion is virtually the same. Doubling the population does not mean doubling the sample. Once populations are very large, what determines precision is mainly the desired margin of error.

What really drives the data requirement is the internal variability of the data, not how many users exist in total.

The three ingredients

To calculate the exact number, we need three ingredients:

• Confidence level: How sure do we want to be? Usually 95% (which corresponds to a Z-score of 1.96). We covered this in the article on confidence intervals.
• Margin of error (E): The maximum error we are willing to accept (e.g., 1% or 0.01).
• Expected proportion (p): The estimate of the conversion rate. If we have no idea, we use 0.5 (50%): it is the case of maximum uncertainty and gives the largest possible sample, therefore the most conservative choice.

The formula for estimating a proportion (such as the Conversion Rate) is:

\( n = \frac{Z^2 \cdot p(1-p)}{E^2} \\ \)

where Z is the normal quantile corresponding to the chosen confidence level (1.96 for 95%).

The figure tells the story better than a thousand words. At first, adding data dramatically improves precision. Then the curve flattens: each new hundred observations reduces the margin of error less and less. It is the law of diminishing returns applied to data. Halving the margin of error (going from E = 3% to E = 1.5%) does not require twice the data, but four times as much. There is no such thing as a large sample in absolute terms: there is only a sample large enough for the question we are trying to ask.

Why? Because the standard error decreases with the square root of the sample size. To halve the margin of error, you don’t just double the sample — you need to quadruple it. This is one of those ideas that truly changes how you think about data.

Sample	Margin of error (approx, 95%)
100	± 10%
400	± 5%
1,600	± 2.5%

These are indicative values (assuming p = 0.5). The point is this: to get half the error, you need four times the sample. The table makes it instantly clear.

If we read the curve from the other side — fix the desired margin of error and ask “how much sample do we need?” — we get another useful figure:

The curve rises slowly for wide margins (from E = 10% to E = 5% requires only a few hundred extra observations), then shoots up: going from E = 2.5% to E = 1% demands over 5,000 additional observations. It is the cost of precision, made visible at a glance.

Let’s Calculate It in R and Python

Let’s do a quick example. We want to estimate the Conversion Rate of a new page with a margin of error of 1% (0.01) and a 95% confidence level (Z = 1.96). To be cautious, we set p = 0.5.

Let’s calculate in R:

Z <- 1.96
p <- 0.5
E <- 0.01

n <- (Z^2 * p * (1-p)) / E^2
print(paste("Required sample size:", round(n)))
# Output: Required sample size: 9604

Let’s verify in Python:

Z = 1.96
p = 0.5
E = 0.01

n = (Z**2 * p * (1-p)) / E**2
print(f"Required sample size: {round(n)}")
# Output: Required sample size: 9604

We need about 9,604 users for that precision. If we accepted a margin of error of 2% (E = 0.02), the number would drop to about 2,401. That is the effect of squaring E in the denominator: cutting the precision requirement in half means dividing the required sample by four. Always keep this in mind when deciding what margin to accept.

A concrete SEO example

A page receives 38 clicks in a week and its CTR goes from 5% to 7%. Has it really improved? With such a small number of impressions, we might be observing nothing but statistical noise. The right question is not “did the CTR increase?” but “do we have enough data to distinguish a real change from random fluctuation?”

Another case. We change the title of a landing page and after three days the CTR increases by 20%. Can we celebrate already? Probably not: first we need to ask whether the collected sample (three days of impressions) is sufficient for a stable estimate.

In both cases, the sampling formula gives us a valuable indication: how much data we need before we can say something meaningful.

But there is another question that anyone working with data faces every day: how much data is worth collecting? Each additional observation has a cost — in time, in resources, in complexity. The statistical problem is not just reaching a certain precision, but finding the point where the increase in precision still justifies the time and resources spent. This is exactly the problem that power analysis and effect size address in A/B testing.

Common mistakes

Some misconceptions keep recurring when talking about sampling:

1. Thinking 1,000 observations are always sufficient. It all depends on variability and required precision. For a tight election race you need tens of thousands of respondents; for estimating the mean of a stable industrial process, a few dozen may suffice.
2. Believing only population size matters. As we saw, once the population is large, what really counts is internal variability.
3. Ignoring the sampling method. A large but biased sample is worse than a small but representative one.
4. Increasing the sample until you find a significant result. This practice, known as p-hacking, is one of the most insidious errors in applied statistics. The more data we collect, the more likely we are to find statistically significant differences even when none exist — because statistical significance also depends on sample size, not just on the real effect. We discussed this when talking about the p-value.

From Estimation to A/B Testing

The formula we have seen so far is for estimating a single proportion. But in the daily practice of CRO the problem is almost always different: comparing two proportions, as in an A/B test.

In that case, the logic is the same, but the formula gets more complicated because two new concepts come into play: Effect Size (the minimum difference we want to detect) and Statistical Power. These are the topics of the next articles in this journey, together with hypothesis testing and power analysis.

To avoid the manual calculation, I have prepared an interactive sample size calculator for A/B tests: it does the heavy lifting for us and also tells us how many days to run the test, given the average page traffic.

FAQ

What is the difference between sampling error and bias?
Sampling error is the random fluctuation that decreases as the sample grows. Bias is a systematic error that does not decrease with more data — in fact, a large but biased sample gives precise but wrong estimates.

Does the formula work for finite populations?
Yes, with a correction: if the sample exceeds 5-10% of the total population, a finite population correction factor applies. In web analytics this is rarely necessary because the population (all visitors) is virtually infinite.

What if I have no idea about the expected proportion p?
Use p = 0.5: it is the maximum uncertainty case and produces the largest possible sample. It is the conservative choice.

Can I use the same formula for an A/B test?
No. An A/B test requires specific formulas involving effect size and statistical power. The formula presented here estimates a single proportion. For A/B tests, use the dedicated calculator.

---

The question “how much data do you need?” is, at bottom, the wrong one. Statistics does not look for a magic number valid for every situation: it seeks a compromise between precision, costs, variability, and the goal of the study. Two experiments may require very different samples while studying the same phenomenon. Data quality, collection method, and the question we are asking matter more than any formula.

So far we have discussed the factors that influence sample size: variability, margin of error, confidence level, representativeness, cost. The next step is turning these considerations into a concrete calculation: how much data do we need to compare two versions of a web page, or two SEO strategies? This is the domain of power analysis, and we will cover it in the next article of this series.

Further Reading

If you want to dive deeper into sampling, the biases that can distort it, and the logic of statistical inference, The Art of Statistics by David Spiegelhalter is the perfect companion. Spiegelhalter devotes illuminating pages to real cases — wrong polls, convenience samples, misleading statistics — and shows how the math of sampling is worth little without careful thought about how data is collected.