statistics Archives - Page 2 of 11

Bayesian Conversion Rate Estimation: how much can we trust limited data

In the article on the foundations of Bayesian statistics, we saw how Bayesian updating works through simulation: generate samples from the prior, simulate data, filter. An intuitive method, but one that runs into a practical limit as soon as data becomes even slightly numerous.
In this article we move to the elegant analytical solution that the Bayesian approach provides for one of the most common problems in marketing analysis: estimating a conversion rate with limited data.

The problem always starts the same way. A small e-commerce store has collected 23 conversions out of 412 sessions. The raw rate is 23/412 ≈ 5.6%. A seemingly precise number. But how much do we trust it? We could be looking at the true 3% or the true 9% — with that sample, we simply do not know. The point estimate “5.6%” says nothing about its own uncertainty.

The peeking problem: why sneaking a look at an A/B test inflates false positives

On 21 January 2015 Optimizely — one of the most widely used A/B testing platforms in the world — switched on a completely new statistical engine for all of its customers, the New Stats Engine.
It wasn’t a technical whim: the old engine, built around a classic fixed-horizon t-test (Fixed Horizon) and developed with statisticians from Stanford, had a flaw that affected anyone who looked at a test’s results before the end. And we look at a test’s results always, before the end.

Optimizely had measured the problem themselves, simulating A/A tests — two identical variants, where by construction neither is better than the other, so any declared “winner” is a false alarm.
According to the figures published by Optimizely, on tests of 5,000 visitors anyone checking the numbers after every visitor saw 57% of A/A tests declare a false winner at least once; checking every 500 visitors the figure dropped to 26%, every 1,000 to 20%. Chilling numbers for a tool that is supposed to help us decide with rigour. The rewrite — sequential inference plus false discovery rate control, what they call always-valid — was meant precisely to bring the error, as they put it, “from over 30% to 5%”.

It’s the same deception we ran into closing the article on regression to the mean: there we selected the worst-performing pages — an extreme instant in the space of the data — and let ourselves be fooled by their rebound. Here we select an extreme instant in time: we stop the moment the test proves us right. The mechanism is a cousin, the risk identical.

Regression to the Mean: the SEO Fix That Worked… by Accident

In the Israeli Air Force, Daniel Kahneman recounts, the flight instructors were sure of one thing: praising a cadet after an excellent manoeuvre made him worse, scolding him after a terrible one made him better.
They had seen it happen a thousand times in the field, so it had to be true: with pilots, severity works and compliments backfire.
Except it wasn’t true. An exceptional manoeuvre — in either direction — is part skill and part luck; and luck, on the next attempt, doesn’t repeat. After a brilliant flight you tend to drift back toward your own average (and it looks as if the praise hurt), after a disastrous one you drift back toward the average (and it looks as if the scolding helped). The instructors were crediting themselves with an effect that was just regression to the mean.

The very same illusion waits for us every time we look at a site’s data and decide whether one of our changes “worked”.

A/B Testing: How to Run Statistically Valid Experiments (and the Mistakes to Avoid)

Over the previous articles we have looked at how hypothesis testing works and how the two-sample t-test lets us compare two groups rigorously. We have also built confidence intervals, learned to quantify the uncertainty of our estimates, and seen with the Central Limit Theorem why all this works even when the data are not normal.

But there is one question that, in the day-to-day reality of anyone doing SEO and marketing, comes up almost daily: which variant performs better? Which title tag brings more clicks? Which landing page converts more? Which meta description draws attention? It is not an academic question: it is the question that separates data-driven decisions from opinions disguised as strategies.

The good news is that we already have all the tools to answer it. A/B testing is nothing more than the direct application of the statistical inference concepts we have built step by step: hypothesis testing, comparison between groups, significance. In this article we put it all together.

PCA (Principal Component Analysis): reduce SEO data complexity without losing insights

SEO and web marketing analysis almost always presents us with the same problem: we have too many metrics and we don’t know which ones really matter. Search volume, CPC, competition, CTR, bounce rate, time on page, conversions by source, average position — the list grows with every new tool we add to our stack.

The problem is not a lack of data: it’s that these metrics are often correlated with each other (more impressions lead to more clicks, which lead to more conversions, which increase costs). When variables move together, most of the information they contain is redundant. Principal Component Analysis (PCA) is the tool we have to cut through this redundancy: it reduces a set of many correlated variables to a few uncorrelated components that capture most of the original information.

In clearer terms: PCA finds the “directions” in which our data vary the most, and lets us project everything else onto those few dimensions. It is like looking at a three-dimensional object from its most informative angle instead of from a random viewpoint.

Category: statistics