Paolo Gironi - notes on data analysis, SEO, statistics and retro computing

Survivorship Bias: Why Success Case Studies Fool Us

We have all read it, perhaps on LinkedIn, complete with an upward-sloping chart: “We rewrote the titles of 300 pages and in three months traffic grew by 38%”.
The case study is honest, the numbers are real, and the temptation to copy the recipe is strong. But there is a question the post does not ask, and it is worth asking on its behalf: how many other companies did the same thing, and how did it go for them?
Whoever rewrote their titles and lost 10% does not write a case study. Nor does whoever saw no change at all. Their silence is not a detail: it is half of the data.

In the article on the p-value we saw that even “significant” results can mislead us. Here the trap is even simpler, and for that very reason more insidious: we only look at those who made it. It is called survivorship bias.

The p-value and Its Misconceptions: What p < 0.05 Really Means

“The new title has a p-value of 0.03: there is a 97% probability that it works.” It is a sentence found in reports, in case studies, sometimes even in textbooks. And it is wrong. Not slightly, not pedantically: it is wrong in a way that can lead us to take half our decisions on the basis of results that do not exist.

On this blog the p-value appears in dozens of articles, from hypothesis testing to A/B testing, from regression to multiple comparisons. It is the most used number in applied statistics, and probably the most misunderstood. It is worth stopping once to look at what it says, and above all at what it does not say.

Multiple Comparisons: Why Out of 200 Queries Some Always «Improve» by Chance

In 2009 a group of neuroscientists put a salmon in a functional MRI scanner, showed it a series of photographs of people in social situations and “asked” it what emotions they were feeling. The analysis found a small area of the brain with statistically significant activity. The catch is that the salmon was dead.
The study was a very serious joke (it won an Ig Nobel in 2012), and its target was not MRI: it was a widespread methodological error. With enough tests, something “significant” turns up anyway. The scanner had compared tens of thousands of small brain volumes, one by one, each at the 5% threshold.

In our line of work the dead salmon has a familiar name: Search Console. We compare this month’s CTR with last month’s across 200 queries, find a dozen “significant” changes and write in the report which queries improved. But how many of those twelve really changed? This is the problem of multiple comparisons, and we have already seen its outline when talking about SEO split testing and peeking.

SEO Split Testing: How to Tell Whether a Change Really Worked

Over the past few weeks I have reworked some of the most-read pages of this blog from top to bottom: new openings, rewritten titles, FAQ sections. In a month I will open Search Console, and the temptation will be the usual one: compare the clicks before with the clicks after.
If they have gone up, will it be thanks to me? And if they have gone down, will I have got everything wrong? A before/after comparison cannot answer either question: in the meantime the season changes, competitors move, the algorithm shifts. One core update in the wrong week and the best work in the world looks like a disaster.

In marketing, the classic answer to this problem is the A/B test: two versions, users split at random, results compared. With SEO, though, there is an insurmountable obstacle, because Google sees only one version of the page. We cannot split the users: we can split the pages. That is the idea behind SEO split testing, and the statistical tool that makes it reliable has a name that sounds like a tongue-twister: difference-in-differences.

Keyword Clustering: grouping thousands of queries with K-means and hierarchical clustering

It happens with every reasonably serious project: you export the keyword list from Search Console or a tool, and you find yourself facing thousands of rows. Three thousand, ten thousand queries. Reading them one by one is unthinkable, and grouping them by hand “by feel” is slow, subjective and impossible to reproduce.
Yet we need that grouping: we want to understand which big families of searches exist in our market, in order to decide where to create content, which pages to build, what to bet on.
The question is: can we let the data reveal the groups, instead of imposing them ourselves? Turning that mountain of queries into a few homogeneous sets is the job of keyword clustering.

We have already tackled a close problem, classifying the intent of a query with Naive Bayes — but there we had an ingredient we lack today: a set of already labelled examples to learn from. Here nobody has handed us the labels. This is the territory of clustering, one of the most used tools of machine learning, and in this article we build it in R with its two classic algorithms: K-means and hierarchical clustering.