{"id":3248,"date":"2024-03-24T14:30:46","date_gmt":"2024-03-24T13:30:46","guid":{"rendered":"https:\/\/www.gironi.it\/blog\/?p=3248"},"modified":"2026-03-06T09:17:43","modified_gmt":"2026-03-06T08:17:43","slug":"guide-to-statistical-tests-for-a-b-analysis","status":"publish","type":"post","link":"https:\/\/www.gironi.it\/blog\/en\/guide-to-statistical-tests-for-a-b-analysis\/","title":{"rendered":"Guide to Statistical Tests for A\/B Analysis"},"content":{"rendered":"\n

Statistical tests<\/strong> are fundamental tools for data analysis and informed decision-making. Choosing the appropriate test depends on the characteristics of the data, the hypotheses to be tested, and the underlying assumptions.<\/p>\n\n\n\n\n\n\n\n

In this blog, I have separately covered each of the main statistical tests with dedicated articles. It is indeed crucial to understand the applicability conditions of each test to obtain reliable results and correct interpretations.<\/p>\n\n\n\n

What I aim to do in this article is to provide an “overview,” placing side by side the most common tests that can find daily applicability for a multitude of analyses related to the world of web marketing and for effective A\/B tests. This is a first comparative look, which should ideally push for the necessary in-depth study of each individual topic, but which I wanted to accompany with very simple practical examples, in order to stimulate the reader’s curiosity.<\/p>\n\n\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\tThe Tests We Will Discuss\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
  1. Il Test Z<\/a>
  2. Student's t-Test<\/a>
  3. Welch's t-Test<\/a>
  4. The Chi-square Test<\/a>
  5. Analysis of Variance (ANOVA)<\/a>
  6. Mann-Whitney U Test<\/a>
  7. Fisher's Exact Test<\/a>
  8. An Overview: a Table<\/a><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n

    Il Test Z<\/h3>\n\n\n\n

    The Z test is a statistical hypothesis test used to verify if the sample mean differs significantly from the population mean<\/strong>, when the population variance is known and the sample size is large (usually greater than 30).<\/p>\n\n\n\n

    The Z test applies when the following conditions are met:<\/p>\n\n\n\n

      \n
    • The sample size is large (n > 30) <\/li>\n\n\n\n
    • The population variance is known <\/li>\n\n\n\n
    • The data is approximately normally distributed<\/li>\n<\/ul>\n\n\n\n

      he Z test is used to determine if there is a significant difference between two means of proportions, such as click-through rates. It can be used, for example, to verify if the introduction of a new feature on a website has led to a significant increase in the conversion rate.<\/p>\n\n\n\n

      Example Case:<\/strong> An e-commerce site wants to test if a new version of the shopping cart has improved the conversion rate. The previous conversion rate is 5% with a known variance of 0.0025. After collecting a sample of 500 users, the new observed conversion rate is 6%. Let’s verify if the difference is statistically significant using the Z test.<\/p>\n\n\n\n

      # Original conversion rate
      p0 <- 0.05
      # Original variance
      var0 <- 0.0025
      # Sample size
      n <- 500
      # Observed conversion rate
      p1 <- 0.06

      # Z test calculation
      z <- (p1 - p0) \/ sqrt(var0\/n)
      z
      <\/pre>\n\n\n\n
      [1] 4.472136<\/pre>\n\n\n\n
      The observed z value is 4.47. Assuming a significance level of 0.05, the critical z value is 1.96. Since the observed value is greater than 1.96, we can reject the null hypothesis and conclude that the difference in the conversion rate is statistically significant.<\/p>\n\n\n\n
      Student’s t-Test<\/h3>\n\n\n\n
      Student’s t-test is a statistical hypothesis test used to verify if the mean of a sample differs significantly from a hypothetical value or if two samples have significantly different means. This test applies when the population variance is unknown and the sample size is small (usually less than 30).<\/p>\n\n\n\n
      Student’s t-test applies when the following conditions are met:<\/p>\n\n\n\n
        \n
      • The sample size is small (n < 30) <\/li>\n\n\n\n
      • The population variance is unknown <\/li>\n\n\n\n
      • The data is approximately normally distributed<\/li>\n<\/ul>\n\n\n\n
        Student’s t-test is used to compare the means of two distinct groups, such as the average time spent on the site for users who saw variant A compared to those who saw variant B.
        <\/p>\n\n\n\n
        Example Case: <\/strong>A company wants to test if a new landing page has an impact on the average time spent on the site. An A\/B experiment is conducted with 20 users for each group. The average time spent on the site for the control group is 3 minutes, while for the test group it is 4 minutes. Let’s verify if the difference is statistically significant using Student’s t-test.<\/p>\n\n\n\n
        # Control group data
        control <- c(2.5, 3.1, 2.8, 3.2, 2.9, 3.5, 3.0, 2.7, 3.3, 2.6, 3.4, 3.1, 2.8, 2.9, 3.2, 3.0, 3.1, 2.7, 3.3, 2.8)

        # Test group data
        test <- c(3.8, 4.2, 3.9, 4.1, 4.3, 3.7, 4.5, 4.0, 3.6, 4.2, 4.1, 3.9, 4.3, 3.8, 4.0, 4.2, 3.7, 4.4, 4.1, 3.9)

        # Student's t-test
        t.test(test, control, alternative = \"greater\")

        data:  test and control
        t = 12.585, df = 37.611, p-value = 2.354e-15
        alternative hypothesis: true difference in means is greater than 0
        95 percent confidence interval:
         0.900641      Inf
        sample estimates:
        mean of x mean of y 
            4.035     2.995 <\/pre>\n\n\n\n
        Student’s t-test provides a p-value less than the significance level of 0.05; therefore, we can reject the null hypothesis and conclude that the difference in average time spent on the site between the two groups is statistically significant.<\/p>\n\n\n\n
        Welch’s t-Test<\/h3>\n\n\n\n
        Welch’s t-test is a variant of Student’s t-test that does not require the assumption of equal variances between the two samples. This test applies when the sample sizes and variances are different.<\/p>\n\n\n\n
        Welch’s t-test applies when the following conditions are met:<\/p>\n\n\n\n
          \n
        • The sample sizes are different <\/li>\n\n\n\n
        • The sample variances are different <\/li>\n\n\n\n
        • The data is approximately normally distributed<\/li>\n<\/ul>\n\n\n\n
          Welch’s t-test is used to compare the means of two distinct groups, such as the average income of users who made a purchase on an e-commerce site compared to those who did not make purchases.<\/p>\n\n\n\n
          Example Case<\/strong>: A company wants to test if the average income of users who made a purchase differs from that of users who did not make purchases. An experiment is conducted with 30 users who made a purchase and 20 users who did not make purchases. The average income of users who made a purchase is $50,000, while that of users who did not make purchases is $40,000. Let’s verify if the difference is statistically significant using Welch’s t-test.<\/p>\n\n\n\n
          # Buyers group data
          buyers <- c(48000, 52000, 49000, 51000, 47000, 55000, 53000, 50000, 46000, 54000,
                      49000, 52000, 51000, 48000, 53000, 47000, 54000, 50000, 49000, 52000,
                      48000, 51000, 53000, 47000, 52000, 49000, 50000, 51000, 48000, 53000)

          # Non-buyers group data
          non_buyers <- c(38000, 42000, 39000, 41000, 37000, 43000, 40000, 39000, 42000, 38000,
                          41000, 40000, 39000, 42000, 37000, 41000, 38000, 39000, 40000, 41000)

          # Welch's t-test
          t.test(buyers, non_buyers, alternative = \"greater\", var.equal = FALSE)<\/pre>\n\n\n\n
          Welch Two Sample t-test\n\ndata:  buyers and non_buyers\nt = 17.811, df = 47.626, p-value < 2.2e-16\nalternative hypothesis: true difference in means is greater than 0\n95 percent confidence interval:\n 9556.368      Inf\nsample estimates:\nmean of x mean of y \n    50400     39850 <\/pre>\n\n\n\n
          Welch’s t-test provides a p-value of 2.2e-16. Since this value is less than the significance level of 0.05, we can reject the null hypothesis and conclude that the difference in average income between users who made a purchase and those who did not make purchases is statistically significant.<\/p>\n\n\n\n
          The Chi-square Test<\/h3>\n\n\n\n
          The chi-square test is a non-parametric statistical test used to verify if there is a significant relationship between two categorical variables or if the observed distribution of a categorical variable differs from the expected distribution.<\/p>\n\n\n\n
          The chi-square test applies when the following conditions are met:<\/p>\n\n\n\n
            \n
          • The variables are categorical <\/li>\n\n\n\n
          • The samples are independent <\/li>\n\n\n\n
          • The expected frequencies in each cell of the contingency table are greater than 5<\/li>\n<\/ul>\n\n\n\n
            The chi-square test is used to analyze the association between two categorical variables, such as the relationship between users’ gender and preference for a particular product.
            <\/p>\n\n\n\n
            Example Case:<\/strong> A clothing store wants to understand if there is a relationship between users’ gender and preference for a particular product line. A survey is conducted on 200 users, of which 100 are men and 100 are women. The results show that 60 men and 40 women prefer product line A, while 40 men and 60 women prefer product line B. Let’s verify if there is a significant relationship between gender and preference using the chi-square test.<\/p>\n\n\n\n
            # Observed data
            observed <- matrix(c(60, 40, 40, 60), nrow = 2, byrow = TRUE)
            rownames(observed) <- c(\"Uomini\", \"Donne\")
            colnames(observed) <- c(\"Linea A\", \"Linea B\")
            observed<\/pre>\n\n\n\n
            ##        Line A Line B
            ## Men        60     40
            ## Women      40     60
            <\/pre>\n\n\n\n
            # Chi-square test
            chisq.test(observed)<\/pre>\n\n\n\n
            ## Pearson's Chi-squared test with Yates' continuity correction\n\n## data:  observed\n## X-squared = 7.22, df = 1, p-value = 0.00721\n<\/pre>\n\n\n\n
            The chi-square test provides a p-value of 0.00721. Since this value is less than the significance level of 0.05, we can reject the null hypothesis and conclude that there is a significant relationship between users’ gender and preference for a particular product line.<\/p>\n\n\n\n
            Analysis of Variance (ANOVA)<\/h3>\n\n\n\n
            Analysis of variance (ANOVA) is a statistical test used to compare the means of three or more groups and determine if there are significant differences between them.
            <\/p>\n\n\n\n
            Analysis of variance applies when the following conditions are met:<\/p>\n\n\n\n
              \n
            • The data is approximately normally distributed <\/li>\n\n\n\n
            • The variances of the groups are equal (homoscedasticity<\/em>) <\/li>\n\n\n\n
            • The samples are independent<\/li>\n<\/ul>\n\n\n\n
              Analysis of variance is used to compare the means of different versions of a product, different marketing strategies, or different sales techniques.<\/p>\n\n\n\n
              Example Case:<\/strong> A company wants to test the effectiveness of three different marketing strategies (A, B, and C) on average monthly revenue. 15 stores are selected for each strategy and the average monthly revenue is recorded for a period of 6 months. Let’s verify if there is a significant difference between the marketing strategies using analysis of variance.<\/p>\n\n\n\n
              # Dati
              fatturato_A <- c(120000, 115000, 130000, 125000, 110000, 135000, 118000, 122000, 127000, 115000, 128000, 120000, 124000, 117000, 121000)
              fatturato_B <- c(112000, 118000, 110000, 115000, 122000, 108000, 120000, 114000, 116000, 119000, 111000, 117000, 113000, 121000, 109000)
              fatturato_C <- c(105000, 110000, 108000, 112000, 107000, 115000, 111000, 109000, 113000, 106000, 108000, 114000, 110000, 112000, 107000)

              # Data
              revenue_A <- c(120000, 115000, 130000, 125000, 110000, 135000, 118000, 122000, 127000, 115000, 128000, 120000, 124000, 117000, 121000)
              revenue_B <- c(112000, 118000, 110000, 115000, 122000, 108000, 120000, 114000, 116000, 119000, 111000, 117000, 113000, 121000, 109000)
              revenue_C <- c(105000, 110000, 108000, 112000, 107000, 115000, 111000, 109000, 113000, 106000, 108000, 114000, 110000, 112000, 107000)

              # Analysis of variance
              anova_result <- aov(c(revenue_A, revenue_B, revenue_C) ~ rep(c(\"A\", \"B\", \"C\"), each = 15))
              summary(anova_result)<\/pre>\n\n\n\n
                                               Df    Sum Sq   Mean Sq F value   Pr(>F)    \nrep(c(\"A\", \"B\", \"C\"), each = 15)  2 1.086e+09 543200000    22.7 2.07e-07 ***\nResiduals                        42 1.005e+09  23923810                     \n---\nSignif. codes:  0 \u2018***\u2019 0.001 \u2018**\u2019 0.01 \u2018*\u2019 0.05 \u2018.\u2019 0.1 \u2018 \u2019 1<\/pre>\n\n\n\n
              The analysis of variance provides a p-value of 2.07e-07. Since this value is less than the significance level of 0.05, we can reject the null hypothesis and conclude that there is a significant difference in average monthly revenue between the three marketing strategies.<\/p>\n\n\n\n
              Mann-Whitney U Test<\/h3>\n\n\n\n
              The Mann-Whitney U test is a non-parametric test used to compare the means of two independent groups when the data does not meet the normality or equal variance requirements required for Student’s t-test.<\/p>\n\n\n\n
              The Mann-Whitney U test applies when the following conditions are met:<\/p>\n\n\n\n