With linear regression we learned to predict a number. Here, though, the outcome is a switch, on or off, and a straight line is not the right tool: we’ll see it even ends up predicting negative probabilities. The right tool is logistic regression, one of the most widely used models of all, from marketing to medicine. Let’s build it together, starting from an example with conversions.

In short: logistic regression is the statistical model that estimates the probability of a binary outcome (converts or not, clicks or not) from one or more variables. Instead of a straight line it uses the logistic function, an S-shaped curve that always stays between 0 and 1. The coefficients are expressed in log-odds: exponentiated, they become odds ratios, the factor by which the odds of the event are multiplied when a variable increases by one unit. In R it is fitted with glm(..., family = binomial) and predicted probabilities come from predict(..., type = "response"). To classify, we then need a threshold, which with rare events has to be chosen with care: 0.5 is almost never the right one.

What we’ll cover:

• Why isn’t a straight line enough?
• From probability to odds to logit: the logistic equation
• How do we fit a logistic regression in R?
• How do we read the coefficients? Odds ratios
• From probabilities to decisions: where to put the threshold?
• The conditions for using it
• Frequently asked questions
• Try it yourself
• Further reading

Why isn’t a straight line enough?

Let’s take 5,000 organic sessions on an e-commerce site. For each one we know three things: how many pages it viewed, whether it came from a smartphone or a desktop, and whether the starting query contained the brand name. And we know whether the session ended with a purchase: 271 converted, 5.4%. The data are simulated, so we know the truth and can check whether the model recovers it.

The most immediate idea is to treat conversion as a number, 1 if it happened and 0 if not, and run a least-squares line through it. It even has a name, the linear probability model, and in some cases it works acceptably. But let’s see what happens with our data.

The line says every extra page is always worth 1.9 percentage points of conversion, whether we go from 1 to 2 pages or from 8 to 9. The data tell another story: sessions with few pages convert very little and almost regardless, then the probability speeds up. And when we also add device and brand, things get worse: the linear model assigns 654 sessions out of 5,000 a negative conversion probability, down to −2.2%. A probability below zero has no meaning at all. At the other end, it gives the best profile (12 pages, desktop, brand query) 28.8%, where logistic regression, as we’ll see, estimates 72.9%.

The problem is one of shape. A probability lives between 0 and 1, and near the edges it has to “brake”: it can’t go above 100% or below zero. A straight line, instead, goes on forever. We need a curve with built-in brakes.

From probability to odds to logit: the logistic equation

Logistic regression solves the problem with a change of scale in two steps. The first step is the odds, a word anyone familiar with betting knows well: instead of the probability that an event happens, we look at the ratio between the probability that it happens and the probability that it doesn’t.

\(\text{odds} = \frac{p}{1-p}\)

With a probability of 20%, the odds are 0.2 / 0.8 = 0.25: “one to four”, one success for every four failures. At 50% the odds are 1, and at 80% they rise to 4. The odds have no ceiling at 1: they run from zero to infinity.

The second step is to take their logarithm. The logarithm of the odds, called the logit, also removes the floor at zero and can take any value, from minus infinity to plus infinity. And it is precisely on this unconstrained scale that logistic regression places its linear equation, identical in form to that of multiple regression:

\(\ln\left(\frac{p}{1-p}\right) = b_0 + b_1 x_1 + b_2 x_2 + \dots + b_k x_k\)

where p is the probability of the event, the x’s are the variables (the predictors) and the b’s are the coefficients to estimate. To go back to the probability we just reverse the path, and we get the logistic function, also called the sigmoid because of its S shape:

\(p = \frac{1}{1 + e^{-(b_0 + b_1 x_1 + \dots + b_k x_k)}}\)

Whatever value the expression in brackets takes, the result always stays between 0 and 1. These are the built-in brakes: logistic regression is a straight line on the log-odds scale, which becomes an S-shaped curve on the probability scale.

The coefficients are not found by least squares but by maximum likelihood: among all possible curves, we pick the one that makes the data we actually observed most probable. The calculation is iterative, and we gladly leave it to the software.

How do we fit a logistic regression in R?

Let’s simulate the 5,000 sessions. In real life the data would come from Google Analytics or the CRM; here we build them ourselves, with known effects, so we can check the result:

set.seed(7)
n <- 5000
sessions <- data.frame(
  pages  = 1 + rpois(n, 2.5),    # pages viewed in the session
  mobile = rbinom(n, 1, 0.6),    # 1 = smartphone, 0 = desktop
  brand  = rbinom(n, 1, 0.25)    # 1 = arrived from a query with the brand name
)
true_logit <- -4.4 + 0.35 * sessions$pages - 0.5 * sessions$mobile + 1.2 * sessions$brand
sessions$converts <- rbinom(n, 1, plogis(true_logit))

mean(sessions$converts)
# [1] 0.0542

The plogis() function is the logistic function itself: it turns the true logit into a probability, and rbinom() tosses a loaded coin with that probability for each session. That’s why family = binomial appears in the model: each session is a Bernoulli trial, the building block of the binomial distribution.

Let’s fit the model:

model <- glm(converts ~ pages + mobile + brand, data = sessions, family = binomial)
summary(model)
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept) -4.37566    0.18872 -23.187  < 2e-16 ***
# pages        0.34742    0.03584   9.693  < 2e-16 ***
# mobile      -0.50057    0.12834  -3.900 9.61e-05 ***
# brand        1.19691    0.12868   9.302  < 2e-16 ***
#
#     Null deviance: 2107.0  on 4999  degrees of freedom
# Residual deviance: 1923.8  on 4996  degrees of freedom
# AIC: 1931.8

The model recovers almost perfectly the values we used to build the data: 0.347 against 0.35 for pages, −0.501 against −0.5 for mobile, 1.197 against 1.2 for brand. The columns read as in linear regression: estimate, standard error, the z statistic (here instead of t) and the p-value. The deviance is the logistic counterpart of the residual sum of squares: it drops from 2107 (the model with no predictors) to 1924, a sign that the three variables explain something.

The sign of the coefficients already tells us a lot: more pages and brand queries push the conversion probability up, smartphones push it down. But by how much? Here logistic regression asks for one more step.

How do we read the coefficients? Odds ratios

The coefficients are on the log-odds scale, which nobody is used to thinking in. The trick is to exponentiate them: e raised to the coefficient is the odds ratio, the number by which the odds of conversion are multiplied when the variable increases by one unit, holding the others fixed. Let’s compute them together with their confidence intervals:

round(exp(cbind(OR = coef(model), confint(model))), 3)
#                OR 2.5 % 97.5 %
# (Intercept) 0.013 0.009  0.018
# pages       1.415 1.319  1.519
# mobile      0.606 0.471  0.780
# brand       3.310 2.572  4.261

Let’s translate row by row:

• pages: each extra page viewed multiplies the odds of conversion by 1.415, which means they grow by about 41.5% (with an interval between +32% and +52%);
• mobile: on a smartphone the odds are 60.6% of those on desktop, about 40% lower;
• brand: a session arriving from a query with the brand name has odds 3.3 times those of a non-brand session.

None of the intervals contains 1, the value that would mean “no effect”.

This last point deserves a chart. Let’s compute the predicted probabilities for four session profiles, with type = "response", which returns probabilities instead of log-odds:

profiles <- data.frame(pages  = c(3, 3, 8, 8),
                       mobile = c(1, 0, 1, 0),
                       brand  = c(0, 1, 0, 1))
round(predict(model, newdata = profiles, type = "response"), 3)
#     1     2     3     4
# 0.021 0.106 0.109 0.401

A smartphone session, non-brand, with 3 pages viewed converts with a 2.1% probability. The same number of pages on desktop and from a brand query is worth 10.6%. With 8 pages, desktop and brand, we reach 40.1%.

Let’s check the first value by hand, to see there’s no magic. The logit is −4.376 + 0.347 × 3 − 0.501 × 1 + 1.197 × 0 = −3.834. The probability is 1 / (1 + e^3.834) = 1 / (1 + 46.2) = 0.021.

The four curves share the same odds ratio per page, 1.415. And yet in percentage points the effect of one extra page changes enormously: for a mobile non-brand session, going from 2 to 3 pages adds 0.6 points (from 1.5% to 2.1%); for a mobile session from a brand query, going from 8 to 9 adds 7.6 (from 28.9% to 36.5%). That’s the S-curve at work, and it’s why in reports it pays to show both the odds ratio and a few predicted probabilities for concrete profiles.

From probabilities to decisions: where to put the threshold?

Often we want a decision from the model: which sessions to show an offer to, which users to put into a remarketing campaign. To turn a probability into a yes or a no we need a threshold, and the instinctive choice is 0.5: if the probability is above 50%, we predict a conversion.

To judge the model honestly, though, we can’t evaluate it on the same data we fitted it on. Let’s split the sessions: we fit on 3,500 and measure the predictions on the other 1,500, which the model has never seen.

set.seed(11)
rows <- sample(n, 3500)
train <- sessions[rows, ]
test <- sessions[-rows, ]
model_train <- glm(converts ~ pages + mobile + brand, data = train, family = binomial)
p_test <- predict(model_train, newdata = test, type = "response")

table(predicted = p_test > 0.5, actual = test$converts)
#          actual
# predicted    0    1
#     FALSE 1410   89
#     TRUE     1    0
mean((p_test > 0.5) == test$converts)
# [1] 0.94

94% accuracy: it looks like a great result. But let’s look at the table: of the 89 sessions that actually converted, the model identified none. It simply answered “no” almost every time. And with a 5.9% conversion rate, anyone who always answers “no”, with no model at all, gets 94.1% accuracy, better than us.

With rare events, accuracy is a misleading metric. The model isn’t useless: it’s the threshold that’s wrong. In our data almost no session goes above a 50% probability, so a 0.5 threshold amounts to never flagging anything. Let’s lower it to 0.1:

table(predicted = p_test > 0.1, actual = test$converts)
#          actual
# predicted    0    1
#     FALSE 1292   62
#     TRUE   119   27

Now the model flags 146 sessions and catches 27 that really convert: 30% of conversions (the sensitivity, or recall). Among the flagged sessions, 18.5% convert (the precision), more than three times the starting 5.9%. Accuracy drops to 88%, but the model has become useful.

The chart shows the trade-off. Converting sessions sit further to the right on average, but the two distributions overlap a lot: every threshold is a swap between conversions caught and false alarms. The right threshold isn’t decided by statistics, it’s decided by the cost of the errors: if showing an offer to someone who won’t buy costs little, a low threshold pays off.

To judge the model independently of the threshold, the AUC is often used, the area under the ROC curve: it is the probability that, taking at random one session that converts and one that doesn’t, the model gives the first a higher probability. It’s 0.5 for a model that guesses and 1 for a perfect one. Let’s compute it with ranks, the same idea behind the Wilcoxon test:

r <- rank(p_test)
n1 <- sum(test$converts); n0 <- sum(1 - test$converts)
round((sum(r[test$converts == 1]) - n1 * (n1 + 1) / 2) / (n1 * n0), 3)
# [1] 0.749

An AUC of 0.75 is a fair result: with only three variables, the model orders three pairs out of four correctly.

The conditions for using it

Logistic regression asks for less than linear regression (no normality of residuals, no constant variance), but a few conditions remain:

• the outcome is binary: only two categories (for three or more, see the frequently asked questions below);
• the observations are independent: ten sessions from the same user are not ten independent pieces of information;
• the relationship is linear on the logit scale: if the effect of pages grows up to a point and then flattens out, it has to be modelled (for example with bands of pages viewed);
• the predictors are not too correlated with each other, the multicollinearity problem we’ll discuss in the next article;
• there are enough events: a widespread rule of thumb asks for at least ten or so events (in our case, conversions) per predictor. We have 271 for 3 variables, well above that.

Finally there is a less known condition, and it’s not unusual to read the exact opposite: that logistic regression is used “when the data are linearly separable”. The opposite is true: if a variable perfectly separates the yeses from the noes, logistic regression breaks down. Maximum likelihood pushes the coefficient towards infinity to turn the curve into a perfect step, and the standard error explodes. It happens more often than we’d think, even with famous datasets.

Frequently asked questions

What is the difference between linear and logistic regression?

Linear regression predicts a continuous numeric variable, such as traffic or revenue. Logistic regression predicts the probability of a binary outcome, such as converts or doesn’t convert. To do so it models the logarithm of the odds linearly and returns probabilities that always lie between 0 and 1.

What is the odds ratio in logistic regression?

It is the exponential of a coefficient: it tells by how much the odds of the event are multiplied when the variable increases by one unit, holding the others fixed. An odds ratio of 1.4 means odds 40% higher, one of 0.6 odds 40% lower, one equal to 1 no effect.

How do we get predicted probabilities from a logistic regression?

By applying the logistic function to the linear combination of the coefficients: p = 1 / (1 + e^−(b0 + b1x1 + … + bkxk)). In R, predict(model, newdata, type = "response") returns the probabilities directly; without type = "response" it returns log-odds.

What if there are more than two categories?

If the categories have no order (for example desktop, mobile and tablet), we use multinomial logistic regression, in R with multinom() from the nnet package. If they have a natural order (low, medium, high), we use ordinal logistic regression, in R with polr() from the MASS package.

What is the right threshold for classifying?

It depends on the cost of the two kinds of error and on how frequent the event is. With rare events, such as conversions, a 0.5 threshold often leads to never flagging anything. It’s worth looking at the prediction table for several thresholds, or at the AUC to judge the model independently of the threshold.

How much data does a logistic regression need?

What matters most is the number of events, the cases in the less frequent category. A widespread rule of thumb asks for at least ten or so per predictor: with 50 conversions, better not to go beyond four or five variables.

Try it yourself

The article’s code lends itself to a few experiments.

• Try a 0.2 threshold instead of 0.1: how many conversions does the model catch, and with how many false alarms? Compare the precision with the one at 0.1.
• Remove brand from the model (converts ~ pages + mobile) and look at the AUC on the test set: how much is the information about the starting query worth on its own?
• Raise the true coefficient for pages from 0.35 to 0.7 in the simulation: how does the shape of the curve change, and what happens to the 0.5 threshold?

The last exercise shows well the link between the strength of the predictors and the difficulty of classification: a model with strong effects pushes the probabilities towards the edges, and there one threshold is as good as another.

In our example the three variables are independent by construction: pages viewed don’t depend on the device, brand doesn’t depend on pages. In real life that’s almost never the case. People arriving from a search for the brand name also tend to browse more pages, and when predictors overlap the coefficients become unstable and hard to tell apart. That’s multicollinearity, and together with two other names that sound scarier than they deserve, heteroscedasticity and autocorrelation, it’s the next stop on our journey through regression.

Further reading

To go further into logistic regression and classification, I recommend An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani: its chapter on classification starts from exactly the question of why linear regression doesn’t fit a yes/no outcome, then walks through logistic regression, confusion matrices and thresholds, with R labs that let us reproduce every step.