Surprise: behind the names there are three simple ideas, and above all three very concrete problems. To see them at work, let’s take the model that sooner or later anyone working in digital marketing builds: how many conversions does each euro spent on Google Ads and on Meta bring us? We’ll watch it fall into all three traps, one after the other, and learn how to recognise them and get out.

In short: these are three different problems of regression. Multicollinearity concerns the predictors: two or more explanatory variables are strongly correlated with each other, and the model can’t separate their effects (unstable coefficients, sometimes with the wrong sign). Heteroscedasticity concerns the errors: their spread is not constant, but grows or shrinks with the predicted values. Autocorrelation also concerns the errors, ordered in time: today’s error is linked to yesterday’s. In the last two cases the coefficients remain correct on average, but standard errors, p-values and confidence intervals become unreliable. In R they are diagnosed with the VIF, the Breusch-Pagan test and the Durbin-Watson test.

What we’ll cover:

• The model: how much is each advertising euro worth?
• What is multicollinearity?
• What is heteroscedasticity?
• What is autocorrelation?
• Multicollinearity vs heteroscedasticity vs autocorrelation: what’s the difference?
• Frequently asked questions
• Try it yourself
• Further reading

The model: how much is each advertising euro worth?

Let’s imagine an e-commerce site that for two years has been investing every week in Google Ads and Meta. The business is growing, spend follows the seasons, and in the spreadsheet we have 104 rows: for each week the two budgets and the conversions obtained. The data are simulated, so we know the truth: both channels work, Google a little more than Meta.

Let’s generate the data in R:

library(lmtest)     # Breusch-Pagan and Durbin-Watson tests
library(sandwich)   # robust standard errors

simulate <- function() {
  week <- 1:104
  growth <- seq(5000, 15000, length.out = 104)            # the business grows
  season <- 1 + 0.35 * sin(2 * pi * week / 52)            # high and low season
  google <- round(growth * season * exp(rnorm(104, 0, 0.10)))
  meta <- round(0.45 * google * exp(rnorm(104, 0, 0.08)))  # follows Google
  error <- as.numeric(arima.sim(list(ar = 0.6), 104, sd = 0.12))
  conversions <- round(exp(0.6 + 0.40 * log(google) + 0.20 * log(meta) + error))
  data.frame(week, google, meta, conversions)
}

set.seed(114)
data <- simulate()

The three traps are already written into these lines, even if we can’t see them yet: Meta’s budget is set as a share of Google’s, the error is multiplicative (it grows with the size of the week) and it follows a pattern that drags on from one week to the next. Now let’s do what anyone would do, a regression of conversions on the two kinds of spend:

model <- lm(conversions ~ google + meta, data = data)
summary(model)
# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)
# (Intercept) 179.480922  19.529960   9.190 5.45e-15 ***
# google        0.030642   0.008473   3.616 0.000468 ***
# meta         -0.024130   0.018618  -1.296 0.197920
#
# Multiple R-squared:  0.5323,	Adjusted R-squared:  0.5231
# F-statistic: 57.48 on 2 and 101 DF,  p-value: < 2.2e-16

Let’s read it the way a marketing manager would: every €1,000 on Google brings about 31 conversions. And every €1,000 on Meta takes away 24. Obvious conclusion: switch Meta off.

It would be a costly mistake, because in the data Meta works. The model as a whole explains half of the variability in conversions (R² 0.53, F test with a tiny p-value), but the split between the two channels is an illusion. Welcome to multicollinearity.

What is multicollinearity?

The reason is intuitive. To estimate the effect of Meta “holding Google fixed”, the regression needs weeks in which Meta varies and Google doesn’t. But in our data, when Google goes up Meta goes up too: the correlation between the two kinds of spend is 0.975. The weeks that help tell the two effects apart are very few, and the estimate rests on those few random wobbles.

The standard tool to measure the problem is the VIF, the variance inflation factor. It’s computed with an auxiliary regression: we explain one predictor with all the others and see how well we manage. If the R² of this regression is high, the predictor is almost a copy of the others:

\(VIF = \frac{1}{1 – R^2_{\text{auxiliary}}}\)

The name says exactly what it measures: how many times the variance of the coefficient is inflated compared with the case of independent predictors. Let’s compute it:

r2_aux <- summary(lm(google ~ meta, data = data))$r.squared
r2_aux
# [1] 0.9511464
1 / (1 - r2_aux)
# [1] 20.46933

The car package computes it directly with vif(model). The most widespread rule of thumb considers a VIF above 5 worrying and above 10 serious. Ours is 20.5: the variance of the estimates is twenty times what we’d have with two independent channels, and the standard error (its square root) more than four times.

What does that mean in practice? We see it by repeating the same experiment 500 times: same rules, same true effects, just a different random draw.

set.seed(2026)
estimates <- t(replicate(500, {
  d <- simulate()
  1000 * c(coef(lm(conversions ~ google + meta, data = d))[2:3],
           total = coef(lm(conversions ~ I(google + meta), data = d))[[2]])
}))
mean(estimates[, "meta"] < 0)
# [1] 0.146
cor(estimates[, "google"], estimates[, "meta"])
# [1] -0.9068232

The two estimates compete for the same effect: when one goes up, the other goes down (correlation −0.91 between the estimates). In one case in seven Meta even comes out negative, and our dataset is one of those. And yet the sum of the two effects is stable: total spend has a standard deviation of 2.5 conversions per €1,000, against 8.7 for Google and 17.7 for Meta.

Here is the hallmark of multicollinearity: the model as a whole works, the predictions are good, but the individual coefficients are not credible. It often shows up exactly like this: a fair or high R², a significant F test, and coefficients with huge standard errors or signs that defy common sense.

How do we avoid it in a marketing regression model?

There’s no test that “cures” multicollinearity, because it isn’t a flaw in the calculation: it’s a lack of information in the data. There are four routes.

• Ask a less ambitious question. If the two budgets move together, let’s ask how much total spend is worth. With lm(conversions ~ I(google + meta)) we get 13.6 conversions per €1,000, with a standard error of 1.3: a solid estimate.
• Get the missing information. To separate the channels we need weeks in which they move independently: an experiment where Meta is switched off in some areas or some weeks, the same principle as the control group we use in SEO split tests. More data of the same kind, on the other hand, helps little: if the budgets keep moving together, the problem stays.
• Drop one of the two variables, but carefully: Google would also take credit for Meta, and the estimate would become biased. It’s the same confounding mechanism we saw in multiple regression.
• Use methods that accept some bias in exchange for stability, such as ridge regression, which shrinks the coefficients towards zero. They are the bread and butter of machine learning, but they make interpretation less immediate.

From here on we follow the first route and work with total spend:

data$spend <- data$google + data$meta
model_tot <- lm(conversions ~ spend, data = data)

What is heteroscedasticity?

Well, this term seems made on purpose to scare. But the concept, it turns out, isn’t that complicated.

In marketing, heteroscedasticity is almost the rule. A 300-conversion week can be off by 30 either way; a 600-conversion week, other things equal, is off by 60. Errors grow with size, because effects are proportional rather than additive. The simplest way to notice it is to look at the residuals against the predicted values: if they fan out, the variance isn’t constant.

For a formal check we use the Breusch-Pagan test, which checks whether the squared residuals can be explained by the predictors. The null hypothesis is homoscedasticity:

bptest(model_tot)
# BP = 9.3544, df = 1, p-value = 0.002224

With a p-value of 0.002, the constant-variance hypothesis doesn’t hold.

What happens if we ignore it? Here there’s good news that textbooks often leave in the shade: the coefficients remain correct on average. What breaks is the estimate of their precision: standard errors computed with the classical formula are wrong, and so are p-values and confidence intervals. In our case the robust standard errors, which don’t assume constant variance, are 20% larger (from 0.00131 to 0.00157):

coeftest(model_tot, vcov = vcovHC(model_tot, type = "HC3"))
#               Estimate Std. Error t value  Pr(>|t|)
# (Intercept) 1.7769e+02 2.0789e+01  8.5472 1.310e-13 ***
# spend       1.3622e-02 1.5723e-03  8.6638 7.273e-14 ***

How do we fix it? There are three routes, often combined:

• use robust standard errors, as we just did: the simplest solution, which doesn’t change the estimates but only their uncertainty;
• transform the variables, typically with logarithms, when effects are proportional;
• use weighted least squares, which give less weight to the noisier observations.

In our case the second route is also the most natural, because the error is multiplicative by construction. Let’s move to logarithms:

model_log <- lm(log(conversions) ~ log(spend), data = data)
coef(summary(model_log))
#              Estimate Std. Error    t value     Pr(>|t|)
# (Intercept) 0.2257489 0.45602741  0.4950336 6.216411e-01
# log(spend)  0.5953304 0.04792515 12.4220879 3.942384e-22
bptest(model_log)
# BP = 1.1777, df = 1, p-value = 0.2778

The fan closes, and the test finds nothing anymore. On top of that, the coefficient has gained a very handy interpretation: in a model with logarithms on both sides it’s an elasticity. The value of 0.595 says that 10% more spend brings about 5.8% more conversions (1.1 raised to 0.595 is 1.058). In the simulation the true value is 0.6: the model got very close.

What is autocorrelation?

The last trap remains, the sneakiest one when data are ordered in time.

In marketing it happens all the time: a competitor’s campaign, a period of favourable prices, a Google algorithm update, a good review going round. They’re all things the model doesn’t see and that last more than a week. The errors, then, don’t jump randomly above and below zero: they stay on the same side for weeks on end. The plot of residuals in time order shows it at a glance:

The classical test is Durbin-Watson. Its statistic d is about 2 with no autocorrelation, drops towards 0 with positive autocorrelation and rises towards 4 with negative autocorrelation. In R we no longer need the tables: the test returns the p-value directly.

dwtest(model_log)
# DW = 0.81187, p-value = 2.573e-10
acf(resid(model_log), plot = FALSE)$acf[2]
# [1] 0.5881333

A statistic of 0.81 and a residual autocorrelation of 0.59 between one week and the next: the problem is there, and it’s strong. For longer series or higher-order autocorrelation (between an error and the one two or more periods earlier) we also use the acf() function and the Ljung-Box test, which we met when talking about time series.

What happens if we ignore it? As with heteroscedasticity, the estimates remain correct on average, but the classical standard-error formula assumes that each week brings independent information. With positive autocorrelation that isn’t so: 104 weeks that resemble each other in groups are worth less than 104 independent observations. The model believes it’s more precise than it is. Newey-West standard errors, which account for both autocorrelation and heteroscedasticity, show it right away:

coeftest(model_log, vcov = NeweyWest(model_log))
#             Estimate Std. Error t value  Pr(>|t|)
# (Intercept)  0.22575    1.00072  0.2256     0.822
# log(spend)   0.59533    0.10624  5.6039 1.791e-07 ***

The standard error of the elasticity goes from 0.048 to 0.106, more than double. The 95% confidence interval, which with the classical formula ran from 0.50 to 0.69, widens to 0.38 to 0.81. The effect of advertising remains clearly positive, but our uncertainty about how much it’s worth is more than double what we thought. With a smaller effect, the difference between the two formulas would be the one between a “significant” result and one that isn’t.

How do we fix it?

• use Newey-West standard errors, as above, if what matters is mainly the estimates and their uncertainty;
• model what’s missing: autocorrelation often comes from an omitted variable that moves slowly (seasonality, trend, promotions), and adding it to the model reduces it;
• use models that explicitly estimate the correlation between errors, such as generalised least squares (in R gls() from the nlme package, with correlation = corAR1()). They are the modern descendants of Durbin’s two-step method and of the Cochrane-Orcutt procedure.

Multicollinearity vs heteroscedasticity vs autocorrelation: what’s the difference?

At this point we can put the three concepts side by side. The most important distinction is where the problem lies: multicollinearity concerns the explanatory variables, the other two concern the errors.

There’s a subtle but important difference between the first and the other two. With multicollinearity the standard errors are large and they know it: the model warns us, in its own way, that it can’t separate the effects. With heteroscedasticity and autocorrelation, instead, classical standard errors are wrong silently, and often in the worst direction: they make us look more confident than we are. That’s why diagnostic tests aren’t an appendix for nitpickers, but a check to run every time a regression has to guide a decision, from an advertising ROI model to a forecast of organic traffic.

Frequently asked questions

What is the difference between multicollinearity and autocorrelation?

Multicollinearity concerns the explanatory variables, correlated with each other; autocorrelation concerns the model’s errors, correlated over time. The first makes the coefficients unstable and is measured with the VIF; the second makes the standard errors too optimistic and is checked with the Durbin-Watson test.

What is the difference between heteroscedasticity and multicollinearity?

Heteroscedasticity concerns the errors, whose variance is not constant; multicollinearity concerns the predictors, which are strongly correlated. With heteroscedasticity the coefficients are fine but the classical standard errors are wrong; with multicollinearity the standard errors are honestly large because the data can’t separate the effects.

What is the difference between heteroscedasticity and autocorrelation?

Both concern the model’s errors and leave the coefficients correct on average. Heteroscedasticity is about their spread, which changes from one observation to another; autocorrelation is about their link over time. They can occur together, and Newey-West standard errors account for both.

How do we avoid these problems in a marketing or advertising ROI regression model?

For multicollinearity, avoid moving all channels together, or use experiments that vary one channel at a time; otherwise model total spend. For heteroscedasticity, use logarithms when effects are proportional, or robust standard errors. For autocorrelation in weekly data, add seasonality and trend to the model and use Newey-West standard errors.

What VIF value indicates multicollinearity?

There is no universal threshold. The most widespread rule of thumb considers a VIF above 5 worth watching and above 10 serious. A VIF of 10 means the variance of the coefficient is ten times what it would be with independent predictors.

What should the Durbin-Watson statistic be?

About 2 with no autocorrelation. Values close to 0 indicate positive autocorrelation, values close to 4 negative autocorrelation. In R, dwtest() from the lmtest package also returns the p-value, with no need for tables.

Try it yourself

The article’s code lends itself to a few experiments.

• In the simulate() function, raise the standard deviation of Meta’s noise from 0.08 to 0.40, so the two budgets move more independently. What happens to the VIF? And to the share of negative Meta estimates in the 500 repetitions?
• Set ar = 0 in arima.sim(): does the Durbin-Watson statistic go back close to 2? And do the Newey-West standard errors get close to the classical ones?
• Fit the log model with both channels, lm(log(conversions) ~ log(google) + log(meta)): does multicollinearity disappear with logarithms?

The last exercise has an instructive answer: transformations cure heteroscedasticity, but not multicollinearity. If the two budgets move together, so do their logarithms.

With this we have the tools to trust (or not trust) a regression. It’s time to put them to work on a very concrete SEO problem: building a model of the CTR we expect at each position, and using it to find the pages that perform worse than they should. That’s the topic of expected vs actual CTR, the next stop on our journey.

Further reading

The three problems we discussed are covered in depth in Introduzione all’econometria by Stock and Watson (Italian edition), the book I recommend: it devotes very clear pages to heteroscedasticity-robust standard errors and to Newey-West errors for time series, always starting from concrete economic examples.

Concepts such as the mean or the standard deviation describe one variable at a time. Questions like these, instead, require us to look at two variables together: that is the job of correlation and simple linear regression, two of the most widely used (and most misunderstood) tools in data analysis.

In short: simple linear regression describes the relationship between a dependent variable y and a single independent variable x with a straight line, y = b + m·x, chosen by the least squares method (the line that minimises the sum of squared residuals). The slope m tells us how much y changes on average when x increases by one unit. Pearson’s correlation coefficient r, between −1 and +1, measures the strength and direction of the linear relationship; the coefficient of determination R² is the share of the variability of y explained by the model (with a single predictor, R² = r²). A strong correlation does not prove cause and effect. In R we use cor() and lm().

What Is Simple Linear Regression?

We can consider regression as a method suitable for finding a mathematical relationship that expresses a link between a variable y (the dependent variable or response variable) and a variable x (the independent variable or predictor variable).

The first useful step to qualitatively investigate the possible dependence between two variables x and y is always to draw a graph, called a scatter diagram (scatterplot).

We place the data relating to one of the two variables on the abscissa, the data relating to the other variable on the ordinate, and we represent the individual observations with points. Remember that scatter plots compare two variables.

If there is a simple relationship between the two variables, the diagram should show it!

Let’s use some example values and let R draw our scatter diagram.

I put some imaginary data in a CSV file relating to a hypothetical correlation between the ambient temperature recorded in my city and the sales of ice cream. I name the file gelati.csv and save it as plain text in any folder on my filesystem (in my example in tmp/gelati.csv). The file will have this content:

temperatura,gelati
25,58
30,70
29,61
26,53
25,48
28,66
24,47
22,47
20,40
18,29
22,33

Now I open R Studio and load my dataset:

gelati <- read.csv("/tmp/gelati.csv")

Then I plot the scatterplot to see if the figure is compatible with a linear regression hypothesis:

plot(gelati)

The diagram shows an evident linear correlation, with an upward trend.

How Do We Calculate Pearson’s Correlation Coefficient?

There are various types of correlation coefficients in statistics. If the relationship to be investigated is between two variables of interval or ratio type (i.e., quantitative, numerical variables), the best known and most used is certainly Pearson’s correlation coefficient, generally indicated with the letter r.

I use the R language (not to be confused with the coefficient r) for a practical example with Pearson’s correlation coefficient.
The function to use is cor():

cor(gelati$temperatura,gelati$gelati)

[1] 0.9320279

As you can see, the correlation in this example is very strong.

r is a standardized value ranging from +1 to -1. The further it is from zero and closer to 1 (or -1 in the case of negative correlation), the stronger the correlation.

If r is positive, the correlation is positive, meaning y increases as x increases.
If r is negative, y decreases as x increases.

But when can I say that a correlation is strong or very strong, and when is it moderate or even null? The answer is… it depends 🙂
There is no standard answer. Very arbitrarily, we can say that generally a correlation below 0.20 (or better, between -0.2 and +0.2) is considered very weak, between 0.2 and 0.5 (or between -0.2 and -0.5) moderate, between 0.5 and 0.8 (or between -0.5 and -0.8) quite strong. Correlations higher than 0.8 or lower than -0.8, very strong, are actually quite rare.

CAUTION 1: The most important thing is that the evidence of a relationship between two variables does not necessarily imply the presence of a cause-effect relationship between the two variables. This is a point of utmost importance, which must always be kept in mind. Both variables that in my study show a very strong correlation may actually depend on a third variable, or many other variables, which constitute the real cause. Finding and calculating the correlation between two variables is relatively simple; finding and especially proving a cause-and-effect relationship is extremely complex!

CAUTION 2: Another point I would like to emphasize is that Pearson’s correlation coefficient r measures the linear correlation between two variables. This means that two variables can show an apparent null correlation (r around 0) and yet be correlated, for example showing a curvilinear correlation. A classic textbook example concerns the correlation between stress level and performance in an exam. In fact, a slight stress contributes to improving performance, but beyond a certain threshold it is completely harmful, leading to a decline in the result. In this case, the analysis in terms of r and linear correlation would lead to discarding an existing correlation.

What Does the Coefficient of Determination R² Measure?

By squaring r, we obtain the coefficient of determination.

In our case, the coefficient of determination r² will be:

R-squared = 0.86868

But what does this number mean?

r² tells us to what extent our regression equation reproduces the variance of the data.
In other words, how much of the variation in the response variable is explained by the predictor variable. The more accurate the regression equation, the more the value of r² tends to 1.

With a single predictor, the coefficient of determination is literally the square of r: 0.932² = 0.869. Temperature therefore explains about 87% of the variability in sales, and the remaining 13% comes from everything the model cannot see.
With more predictors the equality no longer holds, and R² grows every time a variable is added, even a useless one: that is why multiple regression looks at the adjusted R².

How Do We Find the Regression Line?

Our goal is to obtain the regression equation, and since it is a linear regression, the typical form will be:

y = mx + b

m indicates the slope of the line in the graph, and is called the regression coefficient.

b is the point where the line intersects the y-axis, and is called the intercept.

Always remember that the linear regression line is the line that best fits the data provided. Ideally, we would like to minimize the distances of all data points from the regression line. These distances are called errors and are also known as residuals. A good line will have small residuals.

We fit the regression line to the data points in a scatter plot using the least squares method.

The calculations are not difficult, and they are worth seeing once. The slope is the ratio between the covariance of x and y and the variance of x; equivalently, it is the correlation r «translated back» into the units of the two variables:

The intercept then follows from requiring the line to pass through the centre of the cloud, that is through the pair of means:

With the numbers already computed in the box on Pearson’s coefficient: m = 0.932 × 13.045 / 3.751 = 3.241 and b = 50.18 − 3.241 × 24.45 = −29.07. In practice, though, nobody does these sums by hand: every advanced scientific calculator and spreadsheet has a dedicated function, and in R the process is even easier.

I proceed by calculating the parameters of the regression line:

# calculate the parameters of the regression line
lm(gelati$gelati ~ gelati$temperatura)

Call: lm(formula = gelati$gelati ~ gelati$temperatura)
Coefficients:
(Intercept) gelati$temperatura
-29.074 3.241

So my line will have the equation:

y = 3.241x - 29.074

Let’s read it: each extra degree is worth, on average, about 3.2 more ice creams sold. And the question we started with gets its answer: at 27 degrees the model predicts 3.241 × 27 − 29.07 ≈ 58 ice creams.
The intercept, on the other hand, should not be taken literally: at zero degrees the line predicts −29 ice creams, which makes no sense. The reason is simple: the data contain no day below 18 degrees, and outside the observed range the line knows nothing. We will come back to this when we talk about extrapolation.

It’s time to plot the scatter diagram again, overlapping the regression line I just found:

# draw the scatterplot
plot(gelati$temperatura,gelati$gelati, main="Scatterplot and Regression Line",xlab="temperature", ylab="gelati")

# and draw the regression line in red
abline(lm(gelati$gelati ~ gelati$temperatura),col="red",lwd=2)

A Case Study: Ad Spend and Conversions

The ice-cream dataset has carried us this far, but let’s now take regression into the territory we actually care about: marketing. We have twelve weeks of Google Ads campaigns, and for each week we know the spend (in euros) and the conversions recorded. The question is the one every client asks us sooner or later: if I increase the budget, how many extra conversions can I expect?

spend,conversions
120,15
150,14
95,11
180,22
210,20
160,18
140,13
250,26
190,17
230,25
110,13
175,16

I fit the model in R:

data <- read.csv("/tmp/ads.csv")
model <- lm(conversions ~ spend, data = data)
summary(model)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  1.93098    2.22949   0.866    0.407
spend        0.09295    0.01284   7.240 2.79e-05 ***

Multiple R-squared:  0.8398,    Adjusted R-squared:  0.8238

Let’s read the results carefully, because every number has something to tell us.

The slope is 0.093: for every extra euro of spend, the model estimates 0.093 additional conversions. In plainer terms: one conversion roughly every 11 euros. Its p-value (2.79e-05) tells us the relationship is anything but random.

The intercept is 1.93: that would be the expected conversions at zero spend. But mind its p-value (0.407): it is not statistically different from zero. The model, in other words, does not have enough data to tell us how much we would convert without campaigns — hardly surprising, since there is not a single zero-spend week in the dataset. Predictions outside the observed range (95–250 euros) remain slippery ground.

The R² is 0.84: spend explains 84% of the variability in weekly conversions. The remaining 16% is everything the model cannot see: seasonality, ad quality, competition, rainy days.

At this point, prediction is child’s play. How many conversions at 200 euros of spend? I compute it in R:

predict(model, data.frame(spend = 200))
# 20.52

One warning before taking this number into a meeting. Regression describes an association; it does not prove a cause. If the budget was raised precisely in the high-demand weeks (sales, seasonality), part of that slope belongs to demand, not to the campaigns. Establishing a causal link requires a controlled experiment — which is exactly what A/B testing is for.

Outliers and Influential Points

An outlier is an extreme observation that does not fit the general correlation or regression pattern. In practice, in our graph we will see that such outliers, if any, will be very far from the regression line in the y direction.
The inclusion of an outlier can affect the slope and y-intercept of the regression line.

When examining a scatter plot and calculating the regression equation, it is worth considering whether or not anomalous observations should be included. They could in fact be errors in the data – and then they should be excluded – but also “real” values, and in this case they are information of the utmost importance for the analyst.

But when can we speak of outliers? There is no fixed rule when trying to decide whether or not to include an outlier in the regression analysis. This decision depends on the sample size, how extreme the outlier is, and the normality of the distribution.

For univariate data, an empirical rule based on the interquartile range IQR can be used to determine whether or not a point is an outlier.

We proceed in this way:

• Calculate the interquartile range for our data.
• Multiply the interquartile range (IQR) by 1.5.
• Add 1.5 x (IQR) to the third quartile.
  Any number greater than the value found is a suspected extreme datum.
• Subtract 1.5 x (IQR) from the first quartile. Any number lower is a suspected extreme datum.

An influential point is a point that, if removed, produces a significant change in the model estimate. An influential point may or may not be an outlier.

The influence.measures() command provides a whole series of useful influence measures: dfbeta, dffit, covratio, Cook’s distance and leverage points of all observations:

# measure influential points
influence.measures(lm(gelati$gelati ~ gelati$temperatura))

Model Assumptions

For the linear regression model to be effectively usable, some assumptions must be met:

• Linearity: the relationship between X and Y really is a straight line, not a curve.
• Normal distribution of errors: the errors must have, for each value of X, a normal distribution.
• Homoscedasticity: the variability of the errors is constant for each value of X.
• Independence of errors: the errors must be independent for each value of X (it is especially important for observations over time, in which it must be verified that there is no autocorrelation).

The assumptions must always be checked, but they do not all carry the same weight. Without linearity, the line itself is wrong. With heteroscedasticity or serial correlation the coefficient estimates are still right on average, but standard errors, p-values and confidence intervals become unreliable. Non-normal residuals, finally, matter mostly in small samples.
We look at the remedies in the article on regression diagnostics.

Residual Analysis

The residual is equal to the difference between the observed value and the predicted value of Y.

• To estimate the goodness of fit of the regression line to the data, a graphical analysis is appropriate using a scatter plot of the residuals (on the ordinate) and the values of X (on the abscissa).
  
• To verify the assumptions, I can evaluate the graph of the residuals with respect to X: this allows us to establish whether the variability of the errors varies according to the values of X, confirming or not the assumption of homoscedasticity.
  
• To verify linearity, plot the residuals, on the ordinate, with respect to the predicted values, on the abscissa. The points should be distributed symmetrically around a horizontal line with an intercept equal to zero. Different trends indicate the presence of non-linearity.

# look at the distribution of residuals.
# must be balanced above and below the zero line.
lmgelati <- lm(gelati$gelati ~ gelati$temperatura)
plot (lmgelati$residual ~ lmgelati$fitted, ylab="Residuals",
xlab="Fitted")
abline(h=0)

R’s lmtest package provides us with the Breusch-Pagan test to verify the homoscedasticity of the residuals:

# check homoscedasticity of residuals
# using the Breusch-Pagan test
library(lmtest)
testbp <- bptest(gelati ~ temperatura, data=gelati)
testbp

# BP = 0.33208, df = 1, p-value = 0.5644

The p-value of 0.56 gives us no reason to reject the null hypothesis of homoscedasticity: the variability of the residuals looks constant.

As for the normality of the residuals, the frequency histogram allows us to verify or not the condition.

We can also verify the normality of the residuals numerically, using a Shapiro-Wilk test:

# check the normality of the distribution of errors
# with the Shapiro-Wilk test
residui <- residuals(lm(gelati$gelati ~ gelati$temperatura))
shapiro <- shapiro.test(residui)
shapiro

# W = 0.93844, p-value = 0.5023

A check that is often suggested, the t-test on the mean of the residuals, is actually pointless: when the model has an intercept, least squares produces residuals whose mean is exactly zero by construction, whatever the data.

The graph of the residuals with respect to time (and the use of the Durbin-Watson statistic) allows us to highlight the existence or not of autocorrelation.

# Durbin-Watson test for autocorrelation
dwtest(gelati ~ temperatura, data=gelati)

# DW = 1.3644, p-value = 0.07275

The test only makes sense when the observations are in time order (days, weeks): with data that have no natural order, autocorrelation has no meaning. Here the p-value of 0.07 stays above the 5% threshold.

Regression Analysis: Practical Difficulties

Simple regression analysis is a widely used model, but very, very insidious. The generalized tendency, in fact, is to use this type of analysis in a not very conscious and rigorous way – as for example in the simplified example I proposed 🙂
The assumptions underlying the model are rather stringent, and very often ignored…

Frequently, the analysis is carried out without taking into account the way in which these assumptions must be evaluated or the simple linear regression model is chosen instead of more appropriate alternative models.

Another very common error is given by extrapolation: that is, an estimation of values external to the range of observed values is made. This is a big no-no.

The advice is always to start the analysis by looking very carefully at the scatter diagram and to carefully verify the hypotheses underlying the regression model before using the results.

Other Types of Correlation Coefficients

Pearson’s correlation coefficient is certainly the best known, studied and used, but as we have seen it applies in cases where both variables are of quantitative type, measured through an interval or ratio scale. There are other methods that allow us to obtain the measure of correlation between variables of different types. All share the characteristic of being conceptually very similar to Pearson’s r coefficient.

The Point-Biserial Correlation Coefficient

Let’s take the case of an analysis in which one of the variables is of quantitative type (measured on an interval or ratio scale) and the second is a categorical variable with two levels (or dichotomous variable). In this case, the point-biserial correlation coefficient comes to our aid. I will not delve into the concept here, as it is in fact a “special” version of Pearson’s coefficient, leaving the reader to deepen when required by the analysis.

The Phi Coefficient

If we need to know if two dichotomous variables are correlated, we could then resort to the phi coefficient, another “special” case of Pearson’s r coefficient. Many readers will certainly know that two dichotomous variables can also be analyzed using a chi-square test.

Spearman’s Rank Correlation Coefficient Rho (and a Note on Kendall’s Tau)

Here is the formula for Spearman’s coefficient:

where d_i is the difference between the two ranks of the same observation and N the number of observations. r_s can also assume values between –1.00 and +1.00, with the same meanings seen for r.

This shortcut formula is exact only when there are no tied ranks. With ties, Pearson’s correlation is computed directly on the ranks, which is what R does. With many ties or small samples, moreover, another statistic is often preferred: Kendall’s tau coefficient.

In R we do not need different functions: cor() computes all three coefficients, we just specify the method. Computing it in R, on our ice-cream example:

cor(gelati$temperatura, gelati$gelati, method = "pearson")   # 0.932
cor(gelati$temperatura, gelati$gelati, method = "spearman")  # 0.954
cor(gelati$temperatura, gelati$gelati, method = "kendall")   # 0.860

Spearman slightly exceeds Pearson: the relationship is monotonic (higher temperatures almost always go with higher sales) even more than it is linear. Kendall’s tau is lower because it measures something else (how much concordant pairs of observations outnumber discordant ones) and tends to give smaller values than Spearman: the two numbers should not be compared directly.

Try It Yourself

To make the mechanism stick, here is an exercise with made-up but realistic data. From Search Console we extracted ten keywords with their average position and CTR (as a percentage):

position,ctr
1.2,28.4
2.5,19.7
3.1,15.2
4.8,9.8
5.5,8.1
6.9,5.9
8.2,4.2
9.0,3.6
10.5,2.8
12.3,1.9

• Fit the regression line of CTR on position with lm(ctr ~ position).
• Interpret the slope: how many CTR points do we lose for each position dropped?
• Predict the expected CTR at position 7.
• Look at the residuals with residuals(): what pattern do they draw?

To check your results: the slope is about −2.15 (each lost position costs two CTR points), the R² is 0.82, and the prediction at position 7 is a CTR of 8.7%. But the most instructive part is the last one: the residuals are positive at the extremes and negative in the middle. That is the typical signature of a non-linear relationship: CTR does not fall in a straight line — it plunges across the top positions and flattens out afterwards. The line is an honest but coarse approximation, and it is precisely the residuals, as we have seen, that give it away.

Frequently asked questions

What is the difference between correlation and regression?

Correlation measures how much two variables vary together and is symmetric: r between x and y equals r between y and x. Regression instead estimates an equation to predict one variable (the dependent one) from the other, and swapping the roles changes the line.

What does an R² of 0.8 mean?

That the model explains 80% of the variability of the dependent variable; the remaining 20% depends on factors the model does not include, or on chance. A high R² does not guarantee that the model is right, nor that the relationship is causal: the residuals must always be checked.

What is the least squares method?

It is the criterion used to choose the regression line: among all possible lines, the one that minimises the sum of the squared vertical distances (the residuals) between the observed points and the line itself.

Does a strong correlation prove cause and effect?

No. Two variables can be strongly correlated because both depend on a third one, or by pure coincidence. Establishing a causal link requires a controlled experiment, such as an A/B test, or an analysis design built for the purpose.

When should we use Spearman instead of Pearson?

When the data are ordinal (ranks, scores, rankings), when the relationship is monotonic but not linear, or when a few extreme values risk distorting Pearson. Spearman works on ranks and is much less sensitive to outliers.

Simple or multiple linear regression?

Simple regression uses a single predictor, multiple regression two or more. When the variable to predict plausibly depends on several factors (spend, seasonality, device), simple regression risks attributing to a single predictor the effect of the others too.

One variable, after all, rarely tells the whole story: conversions depend on spend, but also on seasonality, device, brand strength. When the predictors become more than one, we enter the territory of multiple regression — the next step on this path.

You might also like

• Multiple Regression Analysis, Explained Simply
• Guide to Statistical Tests for A/B Analysis
• Hypothesis Testing: a Step-by-Step Guide

Further Reading

To study regression with the rigor it deserves — estimation, inference, diagnostics — Introduzione all’econometria by Stock and Watson (Italian edition) is the reference textbook, with constant attention to the interpretation of results.