Category: statistics

When we analyse a dataset with many variables and want to predict a category — whether a user will convert or not, whether a page will reach the top 10, whether a keyword is high or low competition — the first problem is figuring out which variables actually matter and how to combine them.
Decision trees tackle this problem in the most natural way possible: a sequence of yes/no questions that, step by step, separate the data into increasingly homogeneous groups.

The general idea

A decision tree is a structure that makes decisions sequentially. Each internal node asks a question (for instance: “is petal length greater than 2.45 cm?”), and depending on the answer it follows a branch to the next node. Eventually we reach a leaf, which assigns the predicted class.

The power of the idea is that the result is intrinsically interpretable: you can understand why the tree made a particular decision by simply following the path from root to leaf. No black box.

Impurity and split criteria

How does the tree decide which question to ask first? The goal is to find the split that separates the data as “cleanly” as possible. The core concept is impurity: a perfect split is one where all elements in a group belong to the same class.

The most common measure is the Gini index. For a node with \( K \) classes, where \( p_k \) is the proportion of elements in class \( k \):

\( G = \sum_{k=1}^{K} p_k (1 – p_k) \\ \)

The index is 0 when the node is pure (all elements belong to the same class), and reaches its maximum when classes are uniformly distributed.
The tree tries all possible splits on all variables and chooses the one that most reduces the weighted average impurity of the two child nodes.

Other measures exist, such as entropy (used in C4.5 trees), but the idea is the same: minimise uncertainty.

A decision tree in R with the iris data

We build a decision tree using the iris dataset, which contains measurements of sepals and petals for 150 flowers of three species. Our goal is to classify the species based on the measurements.

Let’s prepare the data:

data(iris)
library(rpart)

set.seed(123)
train_idx <- sample(1:nrow(iris), 0.8 * nrow(iris))
train <- iris[train_idx, ]
test  <- iris[-train_idx, ]

tree <- rpart(Species ~ ., data = train, method = "class")

Visualise the tree:

library(rpart.plot)
rpart.plot(tree, type = 2, extra = 104)

The tree is very simple. It starts at the root with a question about petal length. If it is less than 2.45 cm, we can be certain the species is setosa (all 41 training cases go there). If it is greater, we move to the next branch, which separates versicolor from virginica again based on petal length, this time with a threshold of 4.75 cm.

Notice that the tree did not use sepal measurements at all: it found them irrelevant for classification and ignored them automatically. This is an implicit form of variable selection: the tree chooses which metrics matter, and in which order.

An SEO example: predicting top 10 pages

Iris is fine for understanding the mechanism, but let’s try something closer to our daily work. We generate synthetic data for 200 web pages with real SEO metrics — word count, readability, speed, backlinks, images, keyword in title, meta description length — and build a tree that predicts whether a page will reach the top 10.

set.seed(2026)

n <- 200
word_count      <- round(runif(n, 300, 3000))
readability     <- round(runif(n, 20, 80), 1)
page_speed_ms   <- round(runif(n, 500, 8000))
backlinks       <- round(runif(n, 0, 150))
images          <- round(runif(n, 0, 20))
keyword_in_title <- rbinom(n, 1, 0.35)
meta_desc_len   <- round(runif(n, 0, 165))

# simulated target: top_10 based on a combination of factors
score <- (
  (backlinks > 30) * 0.30 +
  (keyword_in_title == 1) * 0.20 +
  (readability > 45 & readability < 70) * 0.15 +
  (word_count > 800 & word_count < 2200) * 0.15 +
  (page_speed_ms < 3000) * 0.10 +
  (meta_desc_len > 110 & meta_desc_len < 155) * 0.10
)
score <- score + rnorm(n, 0, 0.12)
top_10 <- factor(ifelse(score > 0.50, "yes", "no"))

seo_data <- data.frame(word_count, readability, page_speed_ms,
                       backlinks, images, keyword_in_title,
                       meta_desc_len, top_10)

The target variable top_10 is constructed so that it depends on all these factors, each with a different weight — backlinks are the most influential, followed by the presence of the keyword in the title, readability in the right range, and so on. We also add a random component to simulate what we cannot measure.

Train the tree:

library(rpart)
train_idx <- sample(1:n, 0.7 * n)
tree_seo <- rpart(top_10 ~ ., data = seo_data[train_idx, ],
                  method = "class", control = rpart.control(cp = 0.01))

Visualise the resulting tree:

library(rpart.plot)
rpart.plot(tree_seo, type = 2, extra = 104)

The tree tells us something very interesting. The first question — the most discriminant variable — is about backlinks: if they are few, the page can hardly reach the top 10, regardless of other factors. Only pages with enough backlinks move to the next branch, where the tree evaluates readability and keyword presence in the title. This is exactly the kind of hierarchy we would expect from a serious SEO analysis: first authority (backlinks), then on-page quality.

Feature importance confirms this hierarchy:

tree_seo$variable.importance

Backlinks dominate, as expected. Readability and keyword in title come next, while speed and meta description length contribute less. The tree did not select the number of images as relevant — in our synthetic data it was not, and the tree ignored it.

n.b. These data are synthetic: in a real project with real data, the tree would reveal the actual hierarchy of variables for your specific context. The example is meant to show the kind of answer you get, not to establish universal truths.

An important caveat: the tree identifies recurring patterns in observed data, not cause-effect relationships. A variable can be very useful for predicting a page’s behaviour without being its direct cause. For instance, if the tree selects CTR as a discriminant variable, it does not mean that increasing CTR will push the page into the top 10 — it could be that pages already in the top 10 tend to have higher CTR precisely because they are in the top 10. The tree tells us what is correlated, not what is causal.

Evaluating accuracy

Let’s check how it performs on the test data:

pred <- predict(tree, newdata = test, type = "class")
table(pred, test$Species)

The contingency table (confusion matrix) shows how many predictions are correct and how many are not. Accuracy is calculated as:

\( \text{accuracy} = \frac{\text{correct predictions}}{\text{total predictions}} \\ \)

Our tree makes very few mistakes: it is a simple model and the iris data are well separated. In the real world, things are almost always more complex.

Overfitting and pruning

Here is the delicate part. A decision tree can keep splitting until every leaf contains a single element. In that case, accuracy on the training set will be 100%. But the tree will have simply memorised the training dataset instead of learning its regularities.

This phenomenon is called overfitting: the model adapts too closely to the training data, capturing noise and specific details that will not appear in new data.

In R, you can prune with rpart using the cp parameter (complexity parameter), which prevents the tree from making splits that do not reduce error by at least a certain threshold. The printcp(tree) function shows cross-validated error for different tree sizes, helping you choose the cut-off point.

Variable importance

One of the most useful byproducts of a decision tree is feature importance: a measure of how much each variable contributes to the overall reduction in impurity. In rpart:

tree$variable.importance

This tells us, for example, that petal length is by far the most discriminant variable for iris — and sepal measurements count for little or nothing.
In web marketing, a similar analysis can reveal which metrics truly separate high-performing pages from low-performing ones.

From single trees to forests: Random Forest

A single decision tree is unstable: a small change in the input data can produce a completely different tree. To reduce this problem, Random Forests build hundreds of trees on slightly different versions of the data (bootstrap sampling) and average their predictions. The result is much more stable and accurate than a single tree, while retaining much of its interpretability.

If a single tree is like asking one expert for their opinion, a Random Forest is like consulting hundreds of independent experts and going with the majority.

We will cover this in a future article.

FAQ

When should I use a decision tree instead of logistic regression?
When the data have complex interactions between variables (for example, the effect of age on conversion changes depending on the acquisition channel), and when interpretability is the priority.

Can a tree handle numerical and categorical variables together?
Yes, rpart handles both types natively. Categorical variables are internally converted to binary splits.

How deep should a tree be?
It depends on the data. The rule of thumb is to use cross-validation (the plotcp() function in rpart) to find the depth that minimises prediction error on unseen data.

Does the decision tree work for regression too?
Yes. Using method = "anova" instead of "class", the tree predicts numerical values (for example, expected time on page).

Why did the tree only use petal length?
Because in the iris data it is the most discriminant variable: it alone suffices to separate one of the three species and almost separates the other two. The tree, being a greedy algorithm, chooses the variable that gives the best immediate split.

---

From decision trees to gradient descent and dimensionality reduction, there is a common thread: all solve the problem of giving structure to complex data, each with its own approach. Our next step in this journey is gradient descent, the engine that powers much of modern machine learning.

Further reading

If you want to dive deeper into decision trees, An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani is the reference: it covers trees, Random Forests and split criteria with the right balance of intuition and formalism. The R labs are freely available online.