{"id":3311,"date":"2021-07-22T15:55:00","date_gmt":"2021-07-22T14:55:00","guid":{"rendered":"https:\/\/www.gironi.it\/blog\/?p=3311"},"modified":"2024-11-14T16:00:10","modified_gmt":"2024-11-14T15:00:10","slug":"multiple-regression-analysis-explained-simply","status":"publish","type":"post","link":"https:\/\/www.gironi.it\/blog\/en\/multiple-regression-analysis-explained-simply\/","title":{"rendered":"Multiple Regression Analysis, Explained Simply"},"content":{"rendered":"\n

The phenomena we observe and wish to study in order to deepen our understanding rarely present themselves so simply as to be defined by only two variables: one predictive<\/strong> (independent<\/em>) and one response<\/strong> (dependent<\/em>).<\/p>\n\n\n\n

Therefore, while simple linear regression analysis<\/a> holds fundamental theoretical importance, in practice it provides little more information than simply studying the correlation coefficient.<\/p>\n\n\n\n\n\n\n\n

This is why, in scientific literature, multiple regression<\/strong> plays a predominant role, offering a comprehensive set of tools to explain the variation of our dependent variable for each predictive variable present in the model, as well as for the interactions among the independent variables.<\/p>\n\n\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tWhat We Will Discuss\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. The Multiple Regression Equation<\/a>
  2. What Insights Can I Gain?<\/a>
  3. A Few Prerequisites to Start<\/a>
  4. How to Proceed Practically?<\/a>
  5. Let's Get Started!<\/a>
  6. How valid is my model?<\/a>
  7. Final summary<\/a>
  8. Free resources for further learning<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n

    The Multiple Regression Equation<\/h2>\n\n\n\n

    Let’s start with the multiple regression equation, which is essentially an “expansion” of the simple linear regression equation and has this general form:<\/p>\n\n\n\n\\(\n\ny = a_1 x_1 + a_2 x_2 + … + a_i x_i + b \\\\ \\)\n\n\n\n

    where
    y is the response variable (note: it is single<\/strong>)
    a1,<\/sub>a2<\/sub>…ai<\/sub> are the regression coefficients for the predictive variables
    x1<\/sub>,x2<\/sub>…xi<\/sub> are the predictive variables
    b is the intercept (also single<\/strong>)<\/p>\n\n\n\n

    What Insights Can I Gain?<\/h2>\n\n\n\n

    First, I need to understand how significantly my predictive variables, when combined, are related to the dependent variable, and what proportion of the outcome is explained by the combination of predictive variables used in the model.<\/strong><\/p>\n\n\n\n

    Next, I need to understand how each predictive variable is linked to the dependent variable, while controlling for the other independent variable (assuming, for simplicity, that there are only two predictive variables).<\/p>\n\n\n\n

    Then, I must determine which of my two predictors has the strongest impact in estimating the dependent variable.<\/p>\n\n\n\n

    These and many other insights can be obtained through the tools provided by multiple regression analysis, and these initial steps already demonstrate the power and practical utility of this technique.<\/p>\n\n\n\n

    A Few Prerequisites to Start<\/h2>\n\n\n\n

    Multiple regression analysis is a powerful and widely used technique, but to use it correctly, certain fundamental assumptions must be met:<\/p>\n\n\n\n

      \n
    1. The dependent variable must be measured on an interval or ratio scale, and the predictive variable must also be interval\/ratio or dichotomous.<\/li>\n\n\n\n
    2. There must be a linear relationship between the predictive variable and the dependent variable.<\/li>\n\n\n\n
    3. All variables in the regression analysis should exhibit a normal distribution.<\/li>\n\n\n\n
    4. The predictive variables should not be highly correlated with each other (known as multicollinearity<\/a>). Prediction errors should be independent of each other.<\/li>\n\n\n\n
    5. It is assumed that heteroscedasticity<\/a> is present. In other words, the errors in predicting Y should be approximately the same in magnitude and direction across all levels of X.<\/li>\n<\/ol>\n\n\n\n

      How to Proceed Practically?<\/h2>\n\n\n\n

      My suggestion for gaining proficiency is to proceed step by step. Here\u2019s my recommended approach:<\/p>\n\n\n\n

        \n
      1. Plot scatterplots for each predictive variable against the response variable to assess the presence of a correlation.<\/li>\n\n\n\n
      2. Calculate the multiple regression equation.<\/li>\n\n\n\n
      3. Verify that the assumptions of normality and homoscedasticity are met.<\/li>\n\n\n\n
      4. Analyze the coefficient of determination.<\/li>\n<\/ol>\n\n\n\n

        For each of these steps, R proves to be, as usual, a highly reliable companion, capable of providing the necessary calculations with just a few commands.<\/p>\n\n\n\n

        Let’s Get Started!<\/h2>\n\n\n\n

        Let\u2019s start R and dive into a practical example, keeping it as simple as possible. We\u2019ll only be scratching the surface here. The goal is to lay the foundation for exploring a vast topic, leaving the reader to delve deeper. So, without further ado, let\u2019s pick a sample dataset.
        Among others, R provides Longley\u2019s Economic Regression Data<\/em>, which I\u2019ll use here.
        What is it? It\u2019s a data frame<\/em> containing seven economic variables from 1947 to 1962.<\/p>\n\n\n\n

        Let\u2019s open R Studio, load the dataset, and take a first look:<\/p>\n\n\n\n

        data(longley) dim(longley) head(longley)<\/pre>\n\n\n\n
        Now, let\u2019s examine the metrics. For our example, we\u2019ll use the number of employed individuals as the response variable and the Gross National Product (GNP) and population as the independent predictive variables.<\/p>\n\n\n\n
        Here\u2019s how the scatterplots look:
        <\/p>\n\n\n\n
        plot(longley$Employed, longley$GNP)<\/pre>\n\n\n
        \n
        \"\"
        employees \/ gross national product<\/figcaption><\/figure>\n<\/div>\n\n\n
        plot(longley$Employed, longley$Population)<\/pre>\n\n\n
        \n
        \"Residuals
        employees \/ population<\/figcaption><\/figure>\n<\/div>\n\n\n
        The growing linear correlation is evident. Let\u2019s proceed with multiple regression. We already know the command: it’s lm(). The syntax for multiple predictor variables is as follows:<\/p>\n\n\n\n
        regression = lm(Employed ~ GNP + Population, data=longley)\nsummary(regression)<\/pre>\n\n\n\n
        The result is a wealth of information!<\/p>\n\n\n\n
        Call:\nlm(formula = Employed ~ GNP + Population, data = longley)\n\nResiduals:\n     Min       1Q   Median       3Q      Max \n-0.80899 -0.33282 -0.02329  0.25895  1.08800 \n\nCoefficients:\n            Estimate Std. Error t value Pr(>|t|)    \n(Intercept) 88.93880   13.78503   6.452 2.16e-05 ***\nGNP          0.06317    0.01065   5.933 4.96e-05 ***\nPopulation  -0.40974    0.15214  -2.693   0.0184 *  \n---\nSignif. codes:  0 \u2018***\u2019 0.001 \u2018**\u2019 0.01 \u2018*\u2019 0.05 \u2018.\u2019 0.1 \u2018 \u2019 1\n\nResidual standard error: 0.5459 on 13 degrees of freedom\nMultiple R-squared:  0.9791,\tAdjusted R-squared:  0.9758 \nF-statistic: 303.9 on 2 and 13 DF,  p-value: 1.221e-11<\/pre>\n\n\n\n
        First, let\u2019s note the p-value of the F-Statistic. It’s very small (1.211e-11), highly significant. Therefore, at least one of our predictor variables is statistically significantly related to the outcome variable. So, let\u2019s proceed.<\/p>\n\n\n\n
        Let\u2019s also plot the residuals:<\/p>\n\n\n\n
        plot(longley$Year, summary(regression)$res, type='o')\nabline(h=0, lty=2, col=\"red\", lwd=2)<\/pre>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        Ideally, the sum of the residuals, or the differences between the actual and predicted values, should tend toward zero, or at least be as low as possible. A look at this section of the summary output tells us that this condition is met in this case:<\/p>\n\n\n
        \n
        \"\"
        the median of the residuals is close to zero, isn\u2019t it?<\/figcaption><\/figure>\n<\/div>\n\n\n
        Let\u2019s visually check the assumption of normality for the residuals:<\/p>\n\n\n\n
        hist(resid(regression),main='Histogram of Residuals',xlab='Standardized Residuals',ylab='Frequency')<\/pre>\n\n\n\n
        The resulting plot shows that the normality assumption is met:<\/p>\n\n\n\n
        \"\"
        the residuals meet the normality assumption<\/figcaption><\/figure>\n\n\n\n
        It’s time to look in detail at what the summary output of our regression tells us.<\/p>\n\n\n\n
        The median is close to zero, and the residuals are approximately normally distributed. So we can proceed.<\/p>\n\n\n\n
        The values of the coefficients for our two predictor variables and the intercept can be found in our output:<\/p>\n\n\n
        \n
        \"\"
        R gives us… everything we need!<\/figcaption><\/figure>\n<\/div>\n\n\n
        The regression line is thus:<\/p>\n\n\n\n\\(\n\\\\\ny = 0.06317x_1 – 0.40974 x_2 + 88.9388\n\\\\\n\\)\n\n\n
        We pay attention to the p-values concerning the slope coefficients. The output shows that our independent variables are significant in predicting the value of the dependent variable, with p-values below the standard 0.05 threshold. The gross national product even has a value below the 0.001 level. We can therefore reject the null hypothesis (predictor variables are not significant) and validate our model, observing that GNP is the most \u201creliable\u201d element for estimation compared to the population<\/strong>. The asterisk notation in the output is also very useful, giving us a quick visual insight into the results.<\/p>\n\n\n
        \n
        \"\"
        Watch the asterisks!<\/figcaption><\/figure>\n<\/div>\n\n\n
        We can also ask R to calculate the confidence interval for our model with the command:<\/p>\n\n\n\n
        confint(regression)<\/pre>\n\n\n\n
        In our example, we obtain this output:<\/p>\n\n\n\n
                          2.5 %       97.5 %\n(Intercept) 59.15805807 118.71953854\nGNP          0.04017053   0.08617434\nPopulation  -0.73841447  -0.08107138<\/pre>\n\n\n\n
        How valid is my model?<\/h2>\n\n\n\n
        At this point, a key player we have encountered in previous articles comes into play: the coefficient of determination<\/strong>, or r2<\/sup>.<\/p>\n\n\n\n
        It provides essential information about how close the data points are to our regression line. In practical terms, it indicates what percentage of the “movements” in the dependent variable are explained by our predictor variables. The value is standardized between 0 and 1, and it\u2019s clear that our model becomes more useful the closer this value gets to 1.<\/p>\n\n\n\n
        R, as always remarkable, has already provided us with this useful value in the output. Here it is:<\/p>\n\n\n
        \n
        \"\"
        when two variables explain almost everything\u2026<\/figcaption><\/figure>\n<\/div>\n\n\n
        Okay, but what is the Adjusted R-squared value? This is the value to consider, as it solves a paradox with the r2<\/sup> value, which always increases as the number of variables increases (even if those variables aren\u2019t significant at all). The adjusted R-squared corrects this anomaly, providing a perfectly usable value (always lower than R-squared).<\/p>\n\n\n\n
        Final summary<\/h2>\n\n\n\n
        I’ll keep it short: a high R-squared value and a very low, close-to-zero residual value indicate a good model<\/strong>.<\/p>\n\n\n\n
        This is not everything; in fact, it\u2019s just the beginning. But it\u2019s the first step in mastering a tool like multiple linear regression analysis, which has great practical utility. I hope to leave you curious to explore further and go beyond (and below are some useful free online resources). Good luck!<\/p>\n\n\n\n
        \n\n\n\n
        Free resources for further learning<\/h2>\n\n\n\n