What we’ll cover<\/strong>:<\/p>\n\n\n\n In everyday life we often have to make decisions based on incomplete information. A new title tag generates a few more clicks: is the title actually better, or was it simply a lucky week?<\/p>\n\n\n\n Hypothesis testing<\/strong> is a statistical procedure that allows us to pose a question based on sample information, to reach a data-driven decision.<\/p>\n\n\n\n In clearer and more direct terms: is my experimental finding due to chance? Hypothesis testing is a statistical procedure for verifying whether chance is a plausible explanation of an experimental result.<\/strong><\/p>\n<\/div><\/div>\n\n\n\n The difference between probability<\/strong> and inference<\/strong> is the starting point:<\/p>\n\n\n\n In hypothesis testing we always weigh two hypotheses. The status quo<\/em> is called the null hypothesis<\/strong>, denoted H\u2080. It is the hypothesis that there is no effect, no difference \u2014 that everything is due to chance.<\/p>\n\n\n\n What we do is test the null hypothesis against an alternative hypothesis<\/strong>, denoted H\u2090 \u2014 which is, in general, the hypothesis we believe in or want to prove.<\/p>\n\n\n\n N.B. In many experimental contexts the alternative hypothesis represents the effect we are trying to verify.<\/em><\/p>\n<\/div><\/div>\n\n\n\n We then choose a significance level<\/strong> or alpha level<\/strong>, denoted \u03b1. The common standard is \u03b1 = 0.05, which corresponds to a 95% confidence level<\/strong>. Based on the alpha level we determine one or more critical regions<\/strong>: the areas of the sampling distribution where, if our test statistic falls, we reject the null hypothesis.<\/p>\n\n\n\n If the value we obtain falls in a critical region, we have sufficient statistical evidence to reject H\u2080 in favour of H\u2081.<\/strong><\/p>\n\n\n\n The test result, of course, does not constitute a certainty. Two types of error are possible:<\/p>\n\n\n\n The clearest way to visualise the two types of error is this table:<\/p>\n\n\n\n Calculating the probability of a Type II error is not as straightforward as for a Type I error. We will tackle it shortly, when we discuss statistical power.<\/p>\n\n\n\n The test can be one-tailed<\/strong>, when the alternative hypothesis is directional (e.g. “the mean is greater than” or “the mean is less than”), or two-tailed<\/strong>, when the alternative hypothesis is simply “different from”:<\/p>\n\n\n\n In a one-tailed test, the entire critical region is concentrated on one side of the distribution. In a two-tailed test, the critical region is split into two symmetric tails, each with area \u03b1\/2.<\/p>\n\n\n\n The choice between one-tailed and two-tailed must be made before<\/strong> collecting data, based on the research question. If we only want to know whether B is different<\/em> from A, we use two tails. If we want to know whether B is better<\/em> than A, we use a one-tailed (right) test.<\/p>\n\n\n\n The operational sequence of hypothesis testing can be summarised in five steps:<\/p>\n\n\n\n The choice between normal and t distribution depends on:<\/p>\n\n\n\n n.b. as the sample grows, the t distribution approaches the normal. For n > 30 the difference is negligible.<\/p>\n<\/div><\/div>\n\n\n\n Let’s return to the opening example. We ran an A\/B test on two title tags:<\/p>\n\n\n\n The difference is 25 clicks, or 0.63 percentage points. Is this a real effect or noise?<\/p>\n\n\n\n Let’s follow the five steps.<\/p>\n\n\n\n We set \u03b1 = 0.05 (the standard level).<\/p>\n\n\n\n For proportions, the sampling distribution of the difference under H\u2080 is approximately normal (the CLT guarantees this with n = 4,000). We use the z-test for two proportions<\/strong>.<\/p>\n\n\n\n In R:<\/p>\n\n\n\n Output:<\/p>\n\n\n\n The observed z value is 2.22 \u2014 the difference between the two CTRs lies more than two standard deviations away from the value expected under H\u2080 (zero). The figure shows the position of the observed value relative to the sampling distribution:<\/p>\n\n\n\n The red area to the right of the dashed blue line is the p-value<\/strong>: the probability of observing a difference equal to or greater than the one obtained, assuming H\u2080 is true.<\/p>\n\n\n\n The p-value is approximately 0.013, below the \u03b1 = 0.05 threshold. We reject the null hypothesis<\/strong>: the difference is statistically significant. Variant B produces a higher CTR than variant A, and chance is not a plausible explanation.<\/p>\n\n\n\n Output:<\/p>\n\n\n\n We can also verify the result with R’s built-in function:<\/p>\n\n\n\n The p-value<\/strong> is the probability of obtaining a result equal to or more extreme than the one observed, assuming the null hypothesis is true. A small p-value indicates that the observed result is unlikely under H\u2080, leading us to reject it.<\/p>\n\n\n\n Critical z-values for common significance levels:<\/p>\n\n\n\n
\n\n\n\nWhat is a hypothesis test<\/h2>\n\n\n\n
Null and alternative hypotheses<\/h2>\n\n\n\n
Type I and Type II errors<\/h2>\n\n\n\n
![\"Type](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2024\/03\/typeItypeIIerrors-1024x412.png\")
One-tailed vs two-tailed tests<\/h2>\n\n\n\n
![\"Critical](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/07\/test-regioni-critiche-en-1.png\")
The five-step procedure<\/h2>\n\n\n\n
An SEO example: A\/B test on CTR<\/h2>\n\n\n\n
1. Hypotheses<\/h3>\n\n\n\n
2. Significance level<\/h3>\n\n\n\n
3. Distribution<\/h3>\n\n\n\n
4. Computation<\/h3>\n\n\n\n
# Data\nn <- 4000\nclicks_A <- 127; clicks_B <- 152\n\n# Pooled proportion under H0\np0 <- (clicks_A + clicks_B) \/ (2 * n) # 0.034875\n\n# Standard error of the difference\nse <- sqrt(p0 * (1 - p0) * (1\/n + 1\/n)) # ~0.00410\n\n# Observed z-score\nz_obs <- (clicks_B\/n - clicks_A\/n) \/ se # ~2.22\nz_obs<\/code><\/pre>\n\n\n\n[1] 2.224<\/code><\/pre>\n\n\n\n![\"Z-test](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/07\/test-pvalue-en-1.png\")
5. Conclusion<\/h3>\n\n\n\n
# Compute the p-value (one-tailed right test)\np_val <- 1 - pnorm(z_obs)\np_val<\/code><\/pre>\n\n\n\n[1] 0.0131<\/code><\/pre>\n\n\n\nprop.test(c(127, 152), c(4000, 4000), alternative = \"greater\", correct = FALSE)<\/code><\/pre>\n\n\n\nThe p-value<\/h2>\n\n\n\n
![\"Power](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/07\/test-potenza-en-1.png\")