{"id":3559,"date":"2026-05-06T15:46:09","date_gmt":"2026-05-06T14:46:09","guid":{"rendered":"https:\/\/www.gironi.it\/blog\/?p=3559"},"modified":"2026-05-06T15:57:54","modified_gmt":"2026-05-06T14:57:54","slug":"sampling-and-sample-size-how-much-data-do-you-really-need","status":"publish","type":"post","link":"https:\/\/www.gironi.it\/blog\/en\/sampling-and-sample-size-how-much-data-do-you-really-need\/","title":{"rendered":"Sampling and Sample Size: How Much Data Do You Really Need?"},"content":{"rendered":"\n<div style=\"background-color: #f8f9fa;padding: 20px;border-radius: 8px;margin-bottom: 30px;border-left: 4px solid #4a90e2\">\n<h3 style=\"margin-top: 0\">In this article:<\/h3>\n<ul style=\"margin-bottom: 0\">\n<li><a href=\"#tipi-campionamento\">How to Choose Who to Measure: Types of Sampling<\/a><\/li>\n<li><a href=\"#formula-dimensione\">Sample Size: The Math Behind the Estimation<\/a><\/li>\n<li><a href=\"#esempio-codice\">Let&#8217;s Calculate It in R and Python<\/a><\/li>\n<li><a href=\"#collegamento-ab-test\">From Estimation to A\/B Testing<\/a><\/li>\n<li><a href=\"#bias-errore\">Sampling Error vs Bias<\/a><\/li>\n<li><a href=\"#prova-tu\">Try It Yourself<\/a><\/li>\n<\/ul>\n<\/div>\n\n\n\n<p>In everyday life, as in web analytics, we often have to make decisions based on incomplete information. How much data do I need to understand if this modification to the landing page worked? Are a thousand visits enough? Are ten thousand too many?<\/p>\n\n\n\n<!--more-->\n\n\n\n<p>We can almost never measure the entire population (for example, all future visitors to a site). We have to work on a <strong>sample<\/strong>. And here lies the delicate balance: a sample that is too small leads to wrong conclusions, while one that is unnecessarily large wastes time and resources. So the question becomes: <strong>how much data do we really need?<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"tipi-campionamento\">How to Choose Who to Measure: Types of Sampling<\/h2>\n\n\n\n<p>Before figuring out <em>how much<\/em> data we need, we must understand <em>how<\/em> to collect it. The three main methods are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Simple random sampling:<\/strong> Every user has exactly the same probability of being chosen. It&#8217;s the gold standard, what we try to achieve when we randomize users in an A\/B test.<\/li>\n<li><strong>Stratified sampling:<\/strong> We divide users into groups (e.g., Mobile and Desktop traffic) and randomly sample within each group, respecting the original proportions. It ensures that no important minority is ignored.<\/li>\n<li><strong>Systematic sampling:<\/strong> We choose one user every <em>k<\/em> (e.g., one user every 10). Easy to implement, but tricky when the data hide a cyclicity (imagine sampling one user every 7: if we end up with only Mondays, the estimate will be skewed from the start).<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"formula-dimensione\">Sample Size: The Math Behind the Estimation<\/h2>\n\n\n\n<p>The intuition is straightforward: the smaller the effect we are looking for (or the more variable the data), the more observations we need to distinguish it from background noise. Sounds hard to formalise? It is more linear than it seems.<\/p>\n\n\n\n<p>To calculate the exact number, we need three ingredients:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Confidence level:<\/strong> How sure do we want to be? We usually use 95% (which corresponds to a Z-score of 1.96).<\/li>\n<li><strong>Margin of error (E):<\/strong> The maximum error we are willing to accept (e.g., 1% or 0.01).<\/li>\n<li><strong>Expected proportion (p):<\/strong> The estimated conversion rate. If we have no idea, we use 0.5 (50%): it represents maximum uncertainty and yields the largest possible sample, so it is the most conservative choice.<\/li>\n<\/ul>\n\n\n\n<p>The formula to estimate a proportion (like the Conversion Rate) is:<\/p>\n\n\n\n<p style=\"text-align: center;font-size: 1.2em\"><strong>n = (Z&sup2; &times; p(1 &#8211; p)) \/ E&sup2;<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"esempio-codice\">Let&#8217;s Calculate It in R and Python<\/h2>\n\n\n\n<p>Let&#8217;s run a quick example. We want to estimate the Conversion Rate of a new page with a margin of error of 1% (0.01) and a confidence level of 95% (Z = 1.96). To stay on the safe side, we set p = 0.5.<\/p>\n\n\n\n<p>The examples below are in both R and Python \u2014 pick whichever language feels more familiar.<\/p>\n\n\n\n<p>Let&#8217;s calculate it in R:<\/p>\n\n\n\n<pre><code class=\"language-r\"># Sample size calculation for a proportion\nZ &lt;- 1.96\np &lt;- 0.5\nE &lt;- 0.01\n\nn &lt;- (Z^2 * p * (1-p)) \/ E^2\nprint(paste(\"Required size:\", round(n)))\n# Output: Required size: 9604<\/code><\/pre>\n\n\n\n<p>Let&#8217;s verify it in Python:<\/p>\n\n\n\n<pre><code class=\"language-python\"># Sample size calculation for a proportion\nZ = 1.96\np = 0.5\nE = 0.01\n\nn = (Z**2 * p * (1-p)) \/ E**2\nprint(f\"Required size: {round(n)}\")\n# Output: Required size: 9604<\/code><\/pre>\n\n\n\n<p>As we can see, around 9,604 users are needed to reach that precision. N.B.: if we accepted a margin of error of 2% (E=0.02), the number would collapse to about 2,401. That is the effect of <em>E<\/em> squared in the denominator: halving the precision requirement divides the required sample by four. Worth keeping in mind whenever we decide which margin to accept.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"collegamento-ab-test\">From Estimation to A\/B Testing<\/h2>\n\n\n\n<p>The formula seen so far estimates a single proportion. But in everyday CRO (Conversion Rate Optimization) work the actual problem is almost always a different one: <em>comparing<\/em> two proportions, as in an A\/B test.<\/p>\n\n\n\n<p>In that case the logic is the same, but the formula gets more complex because two new concepts come into play: the <strong>Effect Size<\/strong> (the minimum difference we want to detect) and the <strong>Statistical Power<\/strong>.<\/p>\n\n\n\n<p>To skip the manual calculation, I built an <a href=\"\/blog\/en\/ab-test-sample-size-calculator\/\">interactive A\/B test sample size calculator<\/a>: it does the dirty work and also indicates how many days the test should run, given the page&#8217;s average traffic.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"bias-errore\">Sampling Error vs Bias<\/h2>\n\n\n\n<p>One point worth keeping firmly in mind before closing. Sampling error (the one the formula handles) is inevitable and shrinks as the data grow. But there is a far more insidious enemy, and no formula captures it: <strong>bias<\/strong>.<\/p>\n\n\n\n<p>If we test a page only during the weekend, we might collect a million visits (sampling error practically zero), but the sample will not be representative of weekday users. So: no formula can save a sample that is biased at the source. A thousand observations gathered well beat a million gathered badly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"prova-tu\">Try It Yourself<\/h2>\n\n\n\n<p>A product page receives roughly 10,000 impressions per month on Google, with an observed CTR of 3.5%. We want to estimate the true CTR with a margin of error of 1 percentage point (E = 0.01) and 95% confidence.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Compute the required sample size with the formula above, first using p = 0.5 (conservative case) and then p = 0.035 (observed CTR).<\/li>\n<li>Compare the two results: how much does the data requirement change once we have a reasonable estimate of p?<\/li>\n<li>Given 10,000 impressions per month, how many months are needed to satisfy the conservative estimate?<\/li>\n<li>If we accepted a 2% margin (E = 0.02), how would the collection time change?<\/li>\n<\/ol>\n\n\n\n<p>Hint: in R, a minimal function is enough \u2014 <code>sample_size &lt;- function(Z, p, E) ceiling((Z^2 * p * (1-p)) \/ E^2)<\/code> \u2014 to be called twice with the two values of <em>p<\/em>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<p>Now we know how to collect an adequate sample and how much data we need. One question remains: how do we use that sample to rigorously compare two versions of the same page? This is where <a href=\"\/blog\/en\/guide-to-statistical-tests-for-a-b-analysis\/\">actual A\/B testing<\/a> comes in, and it is the next step of the path.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"further-reading\">Further Reading<\/h3>\n\n\n\n<p>To dig deeper into sampling, the biases that can distort it, and the logic of statistical inference, <a href=\"https:\/\/www.amazon.it\/dp\/8806246623?tag=consulenzeinf-21\" rel=\"nofollow sponsored noopener\" target=\"_blank\"><em>The Art of Statistics<\/em><\/a> by David Spiegelhalter is the most suitable companion. Spiegelhalter devotes illuminating pages to real cases \u2014 flawed polls, convenience samples, misleading figures \u2014 showing how the mathematics of sampling means little without careful thought on <em>how<\/em> the data are collected.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this article: How to Choose Who to Measure: Types of Sampling Sample Size: The Math Behind the Estimation Let&#8217;s Calculate It in R and Python From Estimation to A\/B Testing Sampling Error vs Bias Try It Yourself In everyday life, as in web analytics, we often have to make decisions based on incomplete information. &hellip; <a href=\"https:\/\/www.gironi.it\/blog\/en\/sampling-and-sample-size-how-much-data-do-you-really-need\/\" class=\"more-link\">Leggi tutto<span class=\"screen-reader-text\"> &#8220;Sampling and Sample Size: How Much Data Do You Really Need?&#8221;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","footnotes":""},"categories":[161],"tags":[],"class_list":["post-3559","post","type-post","status-publish","format-standard","hentry","category-statistics"],"lang":"en","translations":{"en":3559,"it":3558},"uagb_featured_image_src":{"full":false,"thumbnail":false,"medium":false,"medium_large":false,"large":false,"1536x1536":false,"2048x2048":false,"post-thumbnail":false},"uagb_author_info":{"display_name":"autore-articoli","author_link":"https:\/\/www.gironi.it\/blog\/author\/autore-articoli\/"},"uagb_comment_info":0,"uagb_excerpt":"In this article: How to Choose Who to Measure: Types of Sampling Sample Size: The Math Behind the Estimation Let&#8217;s Calculate It in R and Python From Estimation to A\/B Testing Sampling Error vs Bias Try It Yourself In everyday life, as in web analytics, we often have to make decisions based on incomplete information.&hellip;","_links":{"self":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts\/3559","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/comments?post=3559"}],"version-history":[{"count":3,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts\/3559\/revisions"}],"predecessor-version":[{"id":3565,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts\/3559\/revisions\/3565"}],"wp:attachment":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/media?parent=3559"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/categories?post=3559"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/tags?post=3559"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}