statistics Archives - Page 2 of 12

Expected vs Actual CTR: finding the pages that earn fewer clicks than their position deserves

Anyone who spends their days inside Search Console knows that little nagging feeling: a page sits steadily in third position, yet the clicks are few, a CTR that looks like it belongs at the bottom of the page.
The question we usually ask is the wrong one: not “how many clicks does it get?”, but the more uncomfortable one — “how many clicks should it get, sitting where it sits?”. Without a benchmark, a 3% CTR tells us nothing: for position 8 it would be excellent, for position 2 a small disaster.
What we are missing, in order to judge, is an expected CTR: the value to compare the actual one against.

We have already seen, talking about correlation, that position and CTR move together along a steep curve; and that the next step — using one variable to predict another — is the job of linear regression.
Here the two threads tie together: we turn that curve into an expected CTR and measure, page by page, how far each one deviates from it. It is the way to stop reading CTRs as absolute numbers and start reading them for what they really are: deviations from a norm.

Naive Bayes: classifying search intent with Bayes’ theorem

In the article on the multi-armed bandit we used Bayes to decide between variants: shifting traffic toward the one converting best while the test is still running. Now we take a step sideways, while staying within the same line of reasoning: instead of choosing between options, we want to classify, that is to attach to each new observation the most probable label given its features.
The concrete case is one that anyone doing SEO knows well: the intent behind a search query. Someone searching “how to bake a cake” wants to learn something; someone searching “buy shoes online” is ready to pull out a credit card. They are two different worlds, and they call for different content: a guide, a tutorial, a glossary for the first; a product page, a price list, a clearly visible call to action for the second. Getting the intent wrong means answering the right question in the wrong way.

The trouble is that queries are many and always new, and classifying them by hand does not scale. We need a method that learns from a handful of labelled examples and then copes on its own with queries it has never seen. The algorithm that does this with almost disarming elegance is Naive Bayes, and — as the name hints — it starts once again from the Bayes’ theorem that has accompanied us throughout this path.

Multi-armed bandit: optimising the variants while the test is still running

In the article on Bayesian A/B testing we compared two variants at a fixed sample size: we collect the data for the whole planned duration, compute the probability that B beats A, and decide. It is a solid method, but it carries a cost that usually goes unmentioned.
That cost is the traffic that, for the entire duration of the test, we keep sending to the worse variant. If halfway through the experiment B is already winning hands down, every visitor assigned to A is a conversion we are probably throwing away. The fixed-sample test makes us pay for the information we gather: to learn which variant is better, we must keep showing the one we suspect to be the worse.

There is a way to cut this bill, and it is called a multi-armed bandit. The idea is to shift traffic adaptively toward the variant that is winning while the test is still running, instead of waiting for the final verdict. In this article we build one with one of the most elegant and practical algorithms, Thompson sampling, which is the natural continuation of the Bayesian reasoning we have followed so far.

Bayesian A/B Testing: not just “whether” B beats A, but “by how much”

In the article on classic A/B testing we saw how to compare two variants with the two-proportion test: we compute a statistic, get a p-value, and decide whether to reject the null hypothesis. It works, and it is the daily bread of anyone running online experiments. But there is a subtle gap between what the p-value tells us and what we actually want to know.
The p-value answers a convoluted question: “if A and B were identical, how unlikely would it be to observe a difference as large as this one?”. The question we care about in practice is a different, far more direct one: what is the probability that B is better than A? And, right after: by how much, and how much can we trust that “how much”?

The Bayesian approach answers both questions natively. In this article we apply it to the comparison of two variants, picking up the thread we left hanging when we estimated the conversion rate of a single variant.

It is the question behind every conversion test with an SEO angle: two versions of the same landing page competing for the same organic traffic, or two title-and-meta wordings judged by their actual versus expected CTR in the SERP. Wherever there are successes over trials — clicks over impressions, sign-ups over visits, conversions over sessions — the reasoning that follows is the same.

Bayesian Conversion Rate Estimation: how much can we trust limited data

In the article on the foundations of Bayesian statistics, we saw how Bayesian updating works through simulation: generate samples from the prior, simulate data, filter. An intuitive method, but one that runs into a practical limit as soon as data becomes even slightly numerous.
In this article we move to the elegant analytical solution that the Bayesian approach provides for one of the most common problems in marketing analysis: estimating a conversion rate with limited data.

The problem always starts the same way. A small e-commerce store has collected 23 conversions out of 412 sessions. The raw rate is 23/412 ≈ 5.6%. A seemingly precise number. But how much do we trust it? We could be looking at the true 3% or the true 9% — with that sample, we simply do not know. The point estimate “5.6%” says nothing about its own uncertainty.