Category: statistics

We are running an A/B test on two title tags for a product page. Variant A gets 127 clicks out of 4,000 impressions (CTR 3.17%), variant B gets 152 clicks out of 4,000 impressions (CTR 3.80%). A difference of 25 clicks. Is this a real lift, or just random noise?

Questions like this — is this difference due to chance? — arise constantly in any data-driven activity. Hypothesis testing is the statistical procedure that allows us to answer with a methodical approach, distinguishing results that reflect a real phenomenon from those that are merely sampling noise.

In the article on hypothesis testing we saw how to check whether a sample mean is compatible with a hypothesis about the population. In the examples we proposed, however, we knew the population standard deviation, sigma — which allowed us to use the normal distribution and the Z-score.

In practice this is a rather rare case. If we do not know the population sigma, or if we are working with small samples, we need a different distribution: the t distribution, or Student’s distribution.

In the article on hypothesis testing we saw how to check whether a sample mean is compatible with a hypothesis about the population. But in practice the most common case is comparing two groups.

Does a new drug lower blood pressure more than a placebo? Does a training course improve test scores? Does a new landing page produce a higher CTR? In each case we have two groups (treatment vs control, before vs after, variant A vs B) and we want to know whether their means differ. The right tool is the two-sample t-test.

In previous articles, we examined how hypothesis testing works and how the t-distribution allows us to work even when we don’t know the population standard deviation. In both cases, we focused on a specific question: “can I reject the null hypothesis, yes or no?”

But there’s another question, equally important, that we ask ourselves constantly in daily practice: what is the approximate value of the parameter I’m estimating? It’s not enough to know whether the mean differs from a certain value; we want to know where it lies, and with what margin of uncertainty.

This is where confidence intervals (often abbreviated as CI) come into play—one of the most useful and, at the same time, most misunderstood tools in all of inferential statistics.

Continue reading “Confidence Intervals: the 95% doesn’t mean what you think (here’s what it really says)”

Throughout this journey, we’ve examined tools to describe data, test hypotheses, and build models. But there’s a question that comes before all others—one that’s too often ignored: are these data reliable?

In any dataset—daily sessions, organic clicks, conversion rates—values that don’t behave like the others can hide. Values that deviate abnormally from the rest of the distribution. In statistics, we call them outliers, or anomalous values.

Let’s make one point clear immediately: an anomalous value isn’t necessarily an error. It can be a measurement error, certainly (a broken tracking tag, a bot inflating sessions). But it can also be the most important signal in the entire dataset: a Google algorithm update, content going viral, a technical issue crushing traffic. The issue isn’t eliminating anomalies—it’s recognizing them and then deciding what to do about them.

In this article, we’ll examine three statistical methods for identifying outliers, from the most intuitive to the most formal. For each, we’ll look at the logic, the limitations, and practical application with R.