A year-over-year chart, organic clicks down 3.6%, and beside it an extremely narrow confidence interval: from −4.7% to −2.5%. The p-value is on the order of one ten-billionth. Everything seems to say that the drop is real, and that the search for its cause must now begin.
And yet the numbers in this example come from a simulation in which, by construction, nothing changed. The drop is noise, and the test did not notice.
The culprit is not the test but an assumption we accepted without checking it: that a day’s clicks are a draw from a Poisson. We used the Poisson in comparing two periods with counts, on requests and form submissions. On daily clicks, on sessions, on impressions, the assumption must be checked before we trust it, and very often it does not hold. The reason has a name, overdispersion, and the fix costs two lines of R.
In short: in a Poisson the variance equals the mean. In daily web data the variance is almost always much larger, because of days of the week, season, spikes and bots. When that happens, intervals built as Poisson come out too narrow and p-values too small. The check is a ratio: the sum of the squared Pearson residuals divided by the degrees of freedom, which should sit around 1. In our example it is 23. The two fixes are the quasi-Poisson, which widens the standard errors by the square root of that factor, and the negative binomial, which adds a parameter for the variance. On the same data, the 3.6% drop that the Poisson took as certain stops being significant with both.
What we will cover:
The Poisson distribution has a property that makes it extremely handy: a single parameter describes everything. If a phenomenon produces on average 165 events a day, its variance is 165, and its typical oscillation is the square root of 165, that is about 13 events.
There is no need to estimate the variability separately: the mean carries it along.
It is a demanding promise, and it holds only if the events are independent and the rate is steady. Think of a site’s organic clicks. Mondays are more than Saturdays, in summer they are fewer than in autumn, a newsletter or an article picked up by a newspaper doubles a day, a bot inflates another. Every day has its own rhythm, and the rhythms differ from one another: the daily count is not a Poisson with a single mean but a mixture of Poissons with different means. And a mixture always varies more than its components.
Let us take an example, built on purpose. We simulate two years of daily clicks, starting 2 January 2023, with a baseline mean of 165 clicks, a day-of-week effect (Saturday and Sunday are worth a third less), a seasonality of ±16% and twelve peak days between double and quadruple the normal level. On top of this structure we add a further random variability, the one that real data pick up from bots, new pages and changing search results. The key point is one: in the mechanism that generates the data there is no period effect at all. The two years (more precisely, two consecutive periods of 365 days) have the same underlying rhythm, and the difference we will see between the first and the second is produced by chance: by the spikes, which land wherever they happen to land, and by the daily variability.
We build the data in R:
library(MASS)
set.seed(2026)
day <- seq.Date(as.Date("2023-01-02"), by = "day", length.out = 730)
later <- as.integer(seq_along(day) > 365) # 0 = last year, 1 = this year
weekday <- as.integer(format(day, "%u")) # 1 = Monday ... 7 = Sunday
simulate <- function(drop) {
day_factor <- c(1.14, 1.15, 1.12, 1.10, 0.96, 0.70, 0.66)
season <- 1 + 0.16 * sin(2 * pi * seq_along(day) / 365 - pi / 2)
spikes <- rep(1, 730)
spikes[sample(730, 12)] <- runif(12, 2, 4) # 12 campaign or peak days
mu <- 165 * ifelse(later == 1, drop, 1) *
day_factor[weekday] * season * spikes
data.frame(clicks = rnbinom(730, mu = mu, size = 30), later, weekday,
spike = spikes > 1)
}
data <- simulate(drop = 1) # in the simulation nothing changes The drop argument will be useful to us later: with drop = 1 the two years have the same rhythm, with drop = 0.88 the second year really loses 12%. The spike column marks the twelve special days: in real data it would correspond to the campaign calendar, and it will come in handy again at the end.
The first look is the simplest: we compare the mean and the variance of the daily counts. If the Poisson were adequate, their ratio, which is called the dispersion index, should sit around 1.
mean(data$clicks)
var(data$clicks)
var(data$clicks) / mean(data$clicks)
tapply(data$clicks, data$later, mean) The mean is 164.5 clicks a day, the variance 3,823. The ratio is 23.2: the variance is twenty-three times what the Poisson promises.
The last line gives us the means of the two years, 167.5 and 161.5 clicks a day, that is the 3.6% drop we started from.
The raw index, however, mixes two things, and we will see this shortly: the variability that comes from a known structure (day of week, season) and the variability that remains even after we account for it. The more careful check is done on the model we will use for the comparison, not on the raw data. We estimate the year-over-year comparison as a Poisson regression, that is a generalised linear model (the same family as logistic regression) in which the logarithm of the expected rate depends on the year.
For each day, the model tells us how many clicks it expected. The gap between observed and expected, divided by the oscillation the Poisson predicts, is called the Pearson residual:
\(r_i = \frac{y_i-\hat{\mu}_i}{\sqrt{\hat{\mu}_i}}\)
where yᵢ are the clicks observed on day i and μ̂ᵢ those expected by the model. The denominator is the square root of the mean because, in a Poisson, that is the standard deviation. If the model is right, each squared residual is worth about 1 on average (a little less, because we estimated the expected means on the same data: this is why we divide by the degrees of freedom and not by the number of days). So their sum divided by the degrees of freedom, which we call the dispersion factor φ, sits around 1:
\(\hat{\varphi} = \frac{1}{n-p} \sum_{i=1}^{n} r_i^2\)
where n is the number of days (730) and p the model’s parameters (2: the intercept and the year). We compute it in R:
m_pois <- glm(clicks ~ later, family = poisson, data = data)
phi <- sum(residuals(m_pois, type = "pearson")^2) / m_pois$df.residual
phi
m_pois$deviance / m_pois$df.residual
pchisq(m_pois$deviance, m_pois$df.residual, lower.tail = FALSE) The dispersion factor is 23.2, practically identical to the raw index (with a model this simple it was to be expected). The second number printed is a variant of the same check, which uses the deviance in place of the Pearson residuals: it gives 20.2, same substance.
The third is a formal test: if the Poisson model were adequate, the deviance would follow approximately a chi-squared distribution with 728 degrees of freedom (the approximation is good with counts as large as these, less so with few events a day). R returns 0: the p-value is so small that it falls outside the machine’s precision, it is not a true zero. n.b.: the test says that that model is wrong, not why. It may be the variance, but also a badly specified mean or days that are not independent of one another.
Now let us look at what the Poisson model tells us about the comparison. A small function turns the year’s coefficient into a percentage change, with its 95% interval and the p-value:
change <- function(m) {
s <- summary(m)$coefficients["later", ]
ci <- s[1] + c(-1, 1) * qnorm(0.975) * s[2]
data.frame(change = round(100 * (exp(s[[1]]) - 1), 2),
from = round(100 * (exp(ci[1]) - 1), 2),
to = round(100 * (exp(ci[2]) - 1), 2),
p = signif(s[[4]], 3))
}
change(m_pois) The result is the one from the opening: −3.57%, with an interval from −4.66% to −2.48% and a p-value of 2.9×10⁻¹⁰. A drop that does not exist, presented with absolute confidence.
The simplest idea is also the oldest: if the variance is φ times that of a Poisson, we keep the model exactly as it is and inflate the standard errors by the square root of φ. The coefficients stay identical, only the uncertainty we attach to them changes. This is the quasi-Poisson, and in R it is enough to change one word:
m_quasi <- glm(clicks ~ later, family = quasipoisson, data = data)
summary(m_quasi)$dispersion
change(m_quasi) The estimated dispersion is 23.2, the same one we computed by hand. The square root of 23.2 is 4.8, and indeed the interval becomes about 4.8 times wider (for simplicity our function uses the normal for all three models; with Student’s t, which summary() uses for the quasi-Poisson p-value, only the second decimal digit would change): from −8.69% to +1.83%, p-value 0.19. The central estimate has not moved, still −3.57%. But now the interval includes zero, and the drop goes back to being what it is: an oscillation compatible with chance.
The price of the quasi-Poisson is that it is not a real probabilistic model: there is no distribution behind it, only a correction of the variance (this is why «quasi»). It has no true likelihood and no AIC, and so models cannot be compared with the usual tools (what remains is the F test between nested models, with anova(m1, m2, test = "F")). For a comparison between two periods, however, it is often all we need.
The alternative road is to change the distribution. The negative binomial can be read as a mixture of Poissons with different rates (precisely, rates distributed according to a gamma). It is not given that daily clicks follow this scheme exactly, but it is a reasonable model for data in which each day has its own rhythm. It has one more parameter, θ, which sets how much the variance exceeds the mean:
\(\mathrm{Var}(Y) = \mu + \frac{\mu^2}{\theta}\)
where μ is the mean. When θ is very large the second term disappears and we return to the Poisson; when θ is small, the variance grows with the square of the mean. In R the model is fitted with glm.nb() from the MASS package:
m_nb <- glm.nb(clicks ~ later, data = data)
m_nb$theta
change(m_nb) θ is 9.1, and the interval is from −8.23% to +1.31%, p-value 0.15. Here too zero is inside, and the drop is not significant.
n.b.: we generated the data with a negative binomial with θ = 30, and yet the model estimates 9. It is not an error. The model knows nothing about day of week or about the spikes, and all that extra variability ends up in the dispersion parameter, which drops to make room for it.
Let us put the three results side by side:
| model | change | 95% interval | p-value |
|---|---|---|---|
| Poisson | −3.57% | from −4.66% to −2.48% | 2.9×10⁻¹⁰ |
| quasi-Poisson | −3.57% | from −8.69% to +1.83% | 0.19 |
| negative binomial | −3.57% | from −8.23% to +1.31% | 0.15 |
Same data, same estimate, two opposite conclusions. The difference is not in the drop but in how much confidence we attach to it.
There is a legitimate objection. We know part of that variance perfectly well: Saturday and Sunday are worth a third less than the other days, and we know it before we even look at the data. It is not «extra» variability, it is structure, and an honest model should include it. Let us try adding day of week as a variable:
m_pois_g <- glm(clicks ~ later + factor(weekday), family = poisson, data = data)
sum(residuals(m_pois_g, type = "pearson")^2) / m_pois_g$df.residual
change(glm(clicks ~ later + factor(weekday), family = quasipoisson, data = data))
change(glm.nb(clicks ~ later + factor(weekday), data = data)) The dispersion factor falls from 23.2 to 17.4, about a quarter less. The intervals tighten a little, from −8.03% to +1.06% for the quasi-Poisson and from −7.59% to +0.09% for the negative binomial, but zero stays inside in both cases.
The lesson has two faces. The first thing to do with overdispersion is to explain it: a structure we genuinely know (day of week, season, holidays, known campaigns), put into the model, takes variance away from the noise and can make the comparison more precise, as happens here. It is not an automatic rule: every extra variable costs degrees of freedom, and a variable that overlaps with the period can muddle the comparison instead of clarifying it. But almost never can we explain all of it. Even with week in the model, the variance stays seventeen times that of a Poisson, and a pure Poisson model would go on seeing drops that are not there.
Finally, note that with week in the model the negative binomial estimates a slightly different drop (−3.83% instead of −3.59%). The reason is that the two models weight the days differently: in the negative binomial the days with many clicks, which have higher variance, count a little less. It is a good reminder that the point estimate depends on the model, not only on the data.
So far overdispersion has seemed only a way to switch off alarms. Let us see what happens when the drop is real. We regenerate the data with the same mechanism, but this time the second year really loses 12% of its rate:
data_b <- simulate(drop = 0.88)
var(data_b$clicks) / mean(data_b$clicks)
change(glm(clicks ~ later, family = poisson, data = data_b))
change(glm(clicks ~ later, family = quasipoisson, data = data_b))
change(glm.nb(clicks ~ later, data = data_b)) The dispersion index is 30.8, and the estimate of the drop is −13.2% with all three models:
| model | change | 95% interval | p-value |
|---|---|---|---|
| Poisson | −13.2% | from −14.20% to −12.20% | 1×10⁻¹²⁸ |
| quasi-Poisson | −13.2% | from −18.48% to −7.58% | 1.1×10⁻⁵ |
| negative binomial | −13.2% | from −17.67% to −8.49% | 1.5×10⁻⁷ |
All three find the drop, and none gets the direction wrong. But look at the Poisson’s interval: it runs from −14.2% to −12.2%, and the true value, −12%, falls outside it. The other two include it.
So even when the conclusion is right, the Poisson claims a precision we do not have. In a report this becomes «the drop is between 12 and 14%», while the honest sentence is «the drop is between 8 and 18%». If a budget decision depends on that number, the difference counts.
There is however a limit that neither of the two fixes resolves, and it is right to say so. Quasi-Poisson and negative binomial widen the variance of each day, but they go on treating the days as independent of one another. In real data they often are not: a good week tends to be followed by another good week, a technical problem lasts days, not hours. Ignoring this dependence (the autocorrelation) produces intervals that are still too narrow, and widening the variance of each day is not enough to correct it, because the problem lies in the links between one day and the next. In the simulation we excluded it by construction; on real data it is worth looking at the model’s residuals over time, for example with acf(residuals(m, type = "pearson")), before considering the matter closed.
A warning: with overdispersed data a tiny p-value is not a guarantee. Before trusting a test on counts (a poisson.test(), a Poisson glm, an interval built with the square root of the mean) we check that the variance really is close to the mean, and that the days do not depend on one another. On daily clicks, sessions and impressions these are two conditions that rarely hold.
Everything we have seen for counts holds, almost word for word, for proportions. A page’s CTR is the ratio of clicks to impressions, and the natural model is the binomial: every impression is a trial, every click a success. But when we put many pages together, the binomial with a single CTR makes the same demanding promise as the Poisson: that all pages have the same CTR, and that the differences among them are only chance. A product page and a blog post, a page in first position and one in eighth, do not have the same CTR, and no one would expect them to.
Let us take a quick example. We simulate 200 pages, with monthly impressions ranging from 30 to 5,000, and a true CTR that differs from page to page, around 3.5%:
set.seed(2026)
impressions <- round(10 ^ runif(200, log10(30), log10(5000))) # from 30 to 5,000 per page
true_ctr <- rbeta(200, 0.035 * 60, 0.965 * 60) # each page has its CTR
pages <- data.frame(impressions, clicks = rbinom(200, impressions, true_ctr)) Each page’s true CTR comes from a beta distribution, the natural distribution for proportions. The diagnosis is the same as before: a binomial model with a single CTR for all the pages, and the dispersion factor computed on the Pearson residuals:
m_bin <- glm(cbind(clicks, impressions - clicks) ~ 1, family = binomial, data = pages)
sum(residuals(m_bin, type = "pearson")^2) / m_bin$df.residual The factor is 12.4: a strong excess of variability over what the binomial with a single CTR allows (on the scale of the variance, not of the differences between CTRs: in standard-deviation terms the factor is its square root, about 3.5). The first correction is the same as for counts, again changing only one word: quasibinomial in place of binomial. We compute the interval of the overall CTR with the two models:
mean_ctr <- function(m) {
s <- summary(m)$coefficients[1, ]
round(100 * plogis(s[[1]] + c(est = 0, from = -1, to = 1) * qnorm(0.975) * s[[2]]), 2)
}
mean_ctr(m_bin)
mean_ctr(glm(cbind(clicks, impressions - clicks) ~ 1, family = quasibinomial, data = pages)) The estimated CTR is 3.21% in both cases. The binomial interval runs from 3.13% to 3.30%, the quasi-binomial one from 2.92% to 3.53%, about 3.5 times wider (the square root of 12.4).
Here, though, we need a distinction that was not so evident with daily clicks: which uncertainty are we talking about? If the question is «what is the CTR of these 200 pages, with this content and these positions?», the pages are given, and the binomial interval is essentially adequate: the variability from page to page is not noise, it is a fixed feature of the site. If instead the question is «what CTR can we expect from pages like these?», for example to forecast the performance of new pages of the same kind, the variability between pages becomes part of the uncertainty, and the binomial interval is too optimistic. It is the second question for which overdispersion matters, and it is the one we usually ask when we use an average CTR to decide something.
n.b.: the quasi-binomial applies the same dispersion factor to all the pages, but with impressions as different as these (from 30 to 5,000) the excess variability is not the same for all of them: on the large pages the differences in CTR weigh more. The quasi-binomial is therefore a first correction, not the definitive one. The model that accounts for this is the beta-binomial, the equivalent of the negative binomial for proportions: it treats each page’s CTR as drawn from a beta, exactly as we did in the simulation. In R it is not in the base packages, but it can be fitted with dedicated packages or by maximising the likelihood by hand with optim().
There is however a question that the case of proportions makes unavoidable: which average CTR are we estimating? We can divide all the clicks by all the impressions, or compute the CTR of each page and average them:
100 * sum(pages$clicks) / sum(pages$impressions)
100 * mean(pages$clicks / pages$impressions) The first is 3.21%, the second 3.49%. They are not two estimates of the same thing: the first is the site CTR, in which the pages with many impressions weigh more; the second is the typical page CTR, in which each page counts the same. When the pages have different CTRs, the two numbers diverge, and there is no absolutely right way to choose: it depends on the question. «How many clicks do we get from the traffic we have?» wants the first; «what is the CTR of one of our pages, any page?» wants the second. Before comparing the CTR of two groups of pages, it must be decided which of the two is being compared, because with groups of pages of different sizes the two comparisons can give very different results. It is a theme that deserves an article of its own.
An outlier is a single anomalous value; overdispersion is a property of the whole set of data, which all vary more than the model predicts. Removing the peak days reduces overdispersion, but rarely removes it: in our example the spikes are only 12 days out of 730, and the excess variance remains even without them.
To compare two periods they almost always give the same answer, and the quasi-Poisson is simpler to explain: same estimate, wider errors. The negative binomial is preferable when we need predictions or probabilities (for example «how likely is a day below 50 clicks?»), or when we want to compare different models with the AIC, which the quasi-Poisson does not allow.
poisson.test() of the previous article wrong?No, but it must be used with the same check. When the events are independent enough and the rate is stable, as often happens with requests and forms, the Poisson holds. When what moves the numbers are many different causes (campaigns, season, bots), as often happens with clicks and sessions, it does not hold. The type of metric, however, is not enough to decide: the check costs one line, and it is worth always doing it.
The weight of the day of week lessens, but neither the seasonality nor the spikes do. And with fewer observations the dispersion factor is estimated worse. Aggregating is not a cure: it is a way of having less data on which to notice the problem.
We take back the simulated data and remove the twelve spike days, the ones that in reality would correspond to campaigns or mentions in the press. We recognise them by the spike column, not by the number of clicks: in real data the equivalent is the campaign calendar. Simply removing the days with the most clicks would be a dangerous shortcut, because it would cut the tail of the distribution and distort the very variance we want to measure. How far does the dispersion factor fall? And the second year’s drop, does it stay non-significant with the quasi-Poisson?
no_spikes <- data[!data$spike, ]
m <- glm(clicks ~ later + factor(weekday), family = quasipoisson, data = no_spikes)
summary(m)$dispersion
change(m) The dispersion factor falls from 17.4 to 7.9, and the drop (now −2.22%, interval from −5.35% to +1.00%) stays non-significant.
The next step is to add the season as well, for example with a sine and a cosine of period 365 days:
t <- which(!data$spike)
m_s <- glm(clicks ~ later + factor(weekday) + sin(2 * pi * t / 365) + cos(2 * pi * t / 365),
family = quasipoisson, data = no_spikes)
summary(m_s)$dispersion
change(m_s) The dispersion falls again, to 6.0. It is the same order of magnitude as the variability we ourselves put into the simulation with size = 30: for a day with 165 expected clicks, the negative binomial has a variance 1 + 165/30 ≈ 6.5 times the mean (less on the quieter days, more on the busier ones). What remains after we have explained the structure we know is the overdispersion to manage with the model, and it is still six times what the Poisson allows.
Comparing two whole years has an advantage we often forget: the two periods cross the same seasons and almost the same calendar. The problem becomes much more treacherous when we want to assess a specific event, such as a migration or a Google update, and the before and the after are not comparable with each other. There it is not enough to widen the intervals: we need a control group, and it is the subject of the next article, the expected CTR before and after an event.
For the underlying idea of the article — that an under-estimated uncertainty is more dangerous than a declared one — The Signal and the Noise by Nate Silver is a reading worth many manuals. For the code, R for Data Science by Hadley Wickham, Mine Çetinkaya-Rundel and Garrett Grolemund is the practical companion to put the simulations of this article into practice.
The September report is almost ready, and there is one number that does not fit:…
The redesign is approved, the launch date is set, and sooner or later the question…
The annual report lands on the desk with good news on the front page: the…
A site's technical guides climb in the search results, show up more often, and yet…
An annual report often ends with a single line: clicks from Google fell 11.8% compared…
A site receives 612 requests this year and received 640 the year before. The arithmetic…