In short:<\/strong> the p-value<\/strong> is the probability of observing data at least as extreme as those obtained, assuming the null hypothesis is true<\/strong> (that is, that the effect does not exist). It is not the probability that the null hypothesis is true, nor the probability that the result is due to chance. A small p-value does not say the effect is large, and a p-value above 0.05 does not say there is no effect. The 0.05 threshold is a convention, not a natural boundary. To decide, we also need the size of the effect, the confidence interval and an estimate of how many of the ideas we test actually work.<\/p>\n\n\n\n What we will cover<\/strong>:<\/p>\n\n\n\n Let’s start from an A\/B test on two ad headlines: 4,000 impressions each, 200 clicks for headline A (CTR 5.0%) and 240 for headline B (CTR 6.0%). The test on the two proportions returns p = 0.0498.<\/p>\n\n\n\n To read that number we need a small exercise of imagination. Suppose the two headlines are identical<\/strong>, that the CTR difference does not exist and everything depends on the chance of clicks. That is the null hypothesis. In that imaginary world, how likely would we be to see a difference at least as large as the one observed, 40 clicks out of 4,000? The answer is the p-value: about 5%.<\/p>\n\n\n\n As a formula, with H\u2080 the null hypothesis:<\/p>\n\n\n\n \\(p = P(\\text{data at least this extreme} \\mid H_0 \\text{ true})\\)<\/p>\n\n\n\n The vertical bar reads “assuming that”: the whole calculation happens inside<\/em> the null hypothesis. The p-value measures how surprising the data would be if the effect did not exist.<\/strong> It says nothing, directly, about how likely it is that the effect exists. And it is precisely here that almost all the misconceptions are born.<\/p>\n\n\n\n One consequence that few people know, but which helps to understand everything else: when the null hypothesis is true, the p-value is a random number between 0 and 1, with a uniform distribution<\/strong>. A p of 0.03 is exactly as rare as a p of 0.73. We check it by simulating 10,000 A\/B tests in which B changes nothing, and another 10,000 in which B takes the CTR from 5% to 6%:<\/p>\n\n\n\n With no effect, 4.8% of tests fall below 0.05 (the promised 5%, give or take chance) and the quartiles fall at 0.25, 0.50 and 0.75: the distribution is flat. With a real effect, the p-values pile up near zero, but only half fall below 0.05<\/strong>: with 4,000 impressions per variant, the test’s power<\/a> is 50%.<\/p>\n\n\n\n It is the most widespread misconception, and the most costly. A p of 0.03 does not mean there is a 3% probability that the null hypothesis is true, nor 97% that the effect exists. The p-value is the probability of the data assuming H\u2080, not the probability of H\u2080 given the data. It sounds like wordplay, but the two probabilities can be miles apart.<\/p>\n\n\n\n An example from outside statistics. The probability that a person is American, given that they are the President of the United States, is very high. The probability that they are the President, given that they are American, is about one in three hundred and thirty million. Inverting the “given that” changes everything.<\/p>\n\n\n\n To get from the p-value to the question we really care about (does this idea work?) one ingredient is missing: how many of the ideas we put to the test work in the first place<\/strong>. Let’s take a realistic example. We test 1,000 ideas (headlines, meta descriptions, calls to action) and only one in ten really works; the other nine are neutral. I simulate it in R with the same tests as before:<\/p>\n\n\n\n Out of 1,000 tests, 97 come out “significant”. Fifty are ideas that really work; 47 are neutral ideas that passed the threshold by chance<\/strong>. Among our “winners”, almost one in two is a false alarm, and yet each of them had a p-value below 0.05.<\/p>\n\n\n\n The sum can also be done without simulation. If \u03c0 is the share of ideas that work, \u03b1 the threshold and the power the probability of spotting a good idea, the share of false alarms among the significant results is:<\/p>\n\n\n\n \\(\\text{share of false alarms} = \\frac{\\alpha \\, (1 – \\pi)}{\\alpha \\, (1 – \\pi) + \\text{power} \\cdot \\pi}\\)<\/p>\n\n\n\n The numerator is the neutral ideas that pass by chance (5% of the 900); the denominator adds the good ideas the test recognises (50% of the 100). With our numbers: 45 \/ (45 + 50) = 47%<\/strong>. The table shows how the share changes as the two unknowns vary:<\/p>\n\n\n\n
\n\n\n\nWhat is the p-value, in one sentence<\/h2>\n\n\n\n
one_test <- function(ctr_b, n = 4000, ctr_a = 0.05) {\n clicks_a <- rbinom(1, n, ctr_a)\n clicks_b <- rbinom(1, n, ctr_b)\n prop.test(c(clicks_a, clicks_b), c(n, n), correct = FALSE)$p.value\n}\n\nset.seed(1)\np_null <- replicate(10000, one_test(0.05)) # B changes nothing\np_effect <- replicate(10000, one_test(0.06)) # B really works (+20%)\n\nmean(p_null < 0.05)\n# [1] 0.0481\nquantile(p_null, c(0.25, 0.50, 0.75))\n# 25% 50% 75%\n# 0.2483948 0.4970851 0.7524771\nmean(p_effect < 0.05)\n# [1] 0.501<\/code><\/pre>\n\n\n\n![\"The](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/10\/p-value-distribuzione-en.png\")
Misconception 1: “there is a 97% probability that it works”<\/h2>\n\n\n\n
set.seed(5)\nworks <- seq_len(1000) <= 100 # the good ideas are 100 out of 1,000\np_ideas <- sapply(works, function(w) one_test(ifelse(w, 0.06, 0.05)))\n\ntable(significant = p_ideas < 0.05, works)\n\n# works\n# significant FALSE TRUE\n# FALSE 853 50\n# TRUE 47 50<\/code><\/pre>\n\n\n\n![\"A](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/10\/p-value-mille-idee-en.png\")