statistics Archives - paologironi blog

Keyword Clustering: grouping thousands of queries with K-means and hierarchical clustering

It happens with every reasonably serious project: you export the keyword list from Search Console or a tool, and you find yourself facing thousands of rows. Three thousand, ten thousand queries. Reading them one by one is unthinkable, and grouping them by hand “by feel” is slow, subjective and impossible to reproduce.
Yet we need that grouping: we want to understand which big families of searches exist in our market, in order to decide where to create content, which pages to build, what to bet on.
The question is: can we let the data reveal the groups, instead of imposing them ourselves? Turning that mountain of queries into a few homogeneous sets is the job of keyword clustering.

We have already tackled a close problem, classifying the intent of a query with Naive Bayes — but there we had an ingredient we lack today: a set of already labelled examples to learn from. Here nobody has handed us the labels. This is the territory of clustering, one of the most used tools of machine learning, and in this article we build it in R with its two classic algorithms: K-means and hierarchical clustering.

Expected vs Actual CTR: finding the pages that earn fewer clicks than their position deserves

Anyone who spends their days inside Search Console knows that little nagging feeling: a page sits steadily in third position, yet the clicks are few, a CTR that looks like it belongs at the bottom of the page.
The question we usually ask is the wrong one: not “how many clicks does it get?”, but the more uncomfortable one — “how many clicks should it get, sitting where it sits?”. Without a benchmark, a 3% CTR tells us nothing: for position 8 it would be excellent, for position 2 a small disaster.
What we are missing, in order to judge, is an expected CTR: the value to compare the actual one against.

We have already seen, talking about correlation, that position and CTR move together along a steep curve; and that the next step — using one variable to predict another — is the job of linear regression.
Here the two threads tie together: we turn that curve into an expected CTR and measure, page by page, how far each one deviates from it. It is the way to stop reading CTRs as absolute numbers and start reading them for what they really are: deviations from a norm.

Naive Bayes: classifying search intent with Bayes’ theorem

In the article on the multi-armed bandit we used Bayes to decide between variants: shifting traffic toward the one converting best while the test is still running. Now we take a step sideways, while staying within the same line of reasoning: instead of choosing between options, we want to classify, that is to attach to each new observation the most probable label given its features.
The concrete case is one that anyone doing SEO knows well: the intent behind a search query. Someone searching “how to bake a cake” wants to learn something; someone searching “buy shoes online” is ready to pull out a credit card. They are two different worlds, and they call for different content: a guide, a tutorial, a glossary for the first; a product page, a price list, a clearly visible call to action for the second. Getting the intent wrong means answering the right question in the wrong way.

The trouble is that queries are many and always new, and classifying them by hand does not scale. We need a method that learns from a handful of labelled examples and then copes on its own with queries it has never seen. The algorithm that does this with almost disarming elegance is Naive Bayes, and — as the name hints — it starts once again from the Bayes’ theorem that has accompanied us throughout this path.

Multi-armed bandit: optimising the variants while the test is still running

In the article on Bayesian A/B testing we compared two variants at a fixed sample size: we collect the data for the whole planned duration, compute the probability that B beats A, and decide. It is a solid method, but it carries a cost that usually goes unmentioned.
That cost is the traffic that, for the entire duration of the test, we keep sending to the worse variant. If halfway through the experiment B is already winning hands down, every visitor assigned to A is a conversion we are probably throwing away. The fixed-sample test makes us pay for the information we gather: to learn which variant is better, we must keep showing the one we suspect to be the worse.

There is a way to cut this bill, and it is called a multi-armed bandit. The idea is to shift traffic adaptively toward the variant that is winning while the test is still running, instead of waiting for the final verdict. In this article we build one with one of the most elegant and practical algorithms, Thompson sampling, which is the natural continuation of the Bayesian reasoning we have followed so far.

Bayesian A/B Testing: not just “whether” B beats A, but “by how much”

In the article on classic A/B testing we saw how to compare two variants with the two-proportion test: we compute a statistic, get a p-value, and decide whether to reject the null hypothesis. It works, and it is the daily bread of anyone running online experiments. But there is a subtle gap between what the p-value tells us and what we actually want to know.
The p-value answers a convoluted question: “if A and B were identical, how unlikely would it be to observe a difference as large as this one?”. The question we care about in practice is a different, far more direct one: what is the probability that B is better than A? And, right after: by how much, and how much can we trust that “how much”?

The Bayesian approach answers both questions natively. In this article we apply it to the comparison of two variants, picking up the thread we left hanging when we estimated the conversion rate of a single variant.