Have you ever run an A/B test and wondered whether you should use a t-test or a Wilcoxon test? The choice matters: using the wrong test can cause you to miss a significant result — or, worse, make you think a result is significant when it’s not.
Statistical tests fall into two broad families: parametric and non-parametric. The fundamental difference? Parametric tests assume the data follow a known distribution (usually the normal); non-parametric tests make no such assumption. This is not a technical detail — it determines which tests you can use, how powerful they are, and how much you can trust the results.
In this article we’ll look at the differences, when to use each, and how the choice impacts the analysis of your site’s data.
What we’ll cover
- Parametric tests: the power of normality
- Non-parametric tests: flexibility and robustness
- Comparison table
- Try it yourself
- Choosing the right test
Parametric tests: the power of normality
Many parametric tests assume the data are an approximately normally distributed.
The typical conditions for using them are:
- Continuous or interval/ratio data.
- Approximately normal distribution of the data (or of sample means, thanks to the central limit theorem for large samples).
- Homogeneity of variances (homoscedasticity) across groups.
- A sufficiently large sample, when the model assumptions are reasonably met.
The main advantage of parametric tests is higher statistical power: for the same effect size and sample size, a parametric test is more likely to detect a real difference. This is because they leverage the information about the data’s distribution — provided the assumption holds.
Examples of parametric tests we have covered include the Z-test, Student’s t-test, ANOVA (Analysis of Variance), the Pearson correlation coefficient r, and linear regression.
Non-parametric tests: flexibility and robustness
Non-parametric tests (also called distribution-free tests) do not require assuming a particular distribution of the data nor estimating the parameters of such a distribution. They often work on ranks (the ordering of values) rather than on the values themselves.
They fall into two broad categories:
- Goodness-of-fit tests: compare observed frequencies with those expected under a hypothesis. The prime example is the chi-square test.
- Non-parametric alternatives to parametric tests: tests that answer the same question without assuming normality. For example, the Wilcoxon test is the alternative to the t-test for two samples, and Spearman’s rho is the non-parametric alternative to Pearson’s correlation.
Other examples include the Mann-Whitney U test (alternative to the t-test for two independent samples), the Kruskal-Wallis test (alternative to ANOVA), and Kendall’s tau (alternative to Spearman for ties).
The price of this versatility? Non-parametric tests generally have lower statistical power when the assumptions of parametric tests are met. They have lower efficiency: to achieve the same power, you need larger samples.
Comparison table
When to use each? Here is a summary:
| Feature | Parametric tests | Non-parametric tests |
|---|---|---|
| Assumed distribution | Normal (approx.) | None |
| Data type | Continuous, interval/ratio | Any (including ordinal, ranks) |
| Statistical power | Higher (if assumptions hold) | Lower (same sample size) |
| Robustness | Sensitive to outliers, non-normality | Robust to outliers and skewness |
| Minimum sample | Depends on model assumptions | Works even with small n |
| Output | Parameter estimates + p-value | Rank-based test statistic |
| Examples | t-test, ANOVA, Pearson, Z-test | Wilcoxon, Mann-Whitney, Kruskal-Wallis, Spearman |
Try it yourself
An ecommerce site is testing two versions of a product page (A and B). Session times in seconds for 10 visitors each are:
version_A <- c(45, 52, 48, 61, 55, 49, 53, 47, 58, 51)
version_B <- c(42, 65, 44, 70, 50, 68, 55, 71, 49, 63)Let’s run both a t-test (parametric) and a Wilcoxon test (non-parametric) to see if the difference is significant:
# Parametric test (assumes normality)
t.test(version_A, version_B)
# Non-parametric test (rank-based)
wilcox.test(version_A, version_B)Questions:
1. Do the two tests give the same verdict (p < 0.05)?
2. Which test makes more sense given only 10 data points per group?
3. How would the choice change with 100 visitors per version?
Answers:
1. With this particular data, the t-test gives p ≈ 0.12 (not significant), while Wilcoxon gives p ≈ 0.16 (not significant). They agree — but that’s not always the case.
2. With n = 10 it’s hard to verify normality. Wilcoxon is the safer choice: it makes no distributional assumptions and works well with small samples.
3. With n = 100, the central limit theorem guarantees the sample mean is approximately normal even if the original data aren’t. The t-test becomes more justified — and more powerful.
Choosing the right test
The choice between a parametric and a non-parametric test does not depend on which is "best" in absolute terms, but on the nature of the data and the assumptions we are reasonably willing to make.
The rule of thumb is:
- If the data are normal (or the sample is large) and the variances are homogeneous → parametric test (more powerful).
- If the data are not normal, the sample is small, or there are outliers → non-parametric test (more robust).
- When in doubt, run both: if they agree, the answer is robust. If they disagree, investigate why.
Caution: power isn’t everything.
A more powerful parametric test is not always the best choice. If the data violate the assumptions (normality, homoscedasticity), the parametric test’s p-value can be unreliable — exposing you to a false positive or false negative. In such cases, a non-parametric test, though less powerful, gives a more honest result.
For a practical guide to choosing between tests, see the guide to statistical tests for A/B analysis, and for a systematic treatment of assumptions, effect size and power analysis.
Further Reading
For a comprehensive treatment of the choice between parametric and non-parametric tests, with the applicability conditions of each, Statistica by Newbold, Carlson and Thorne offers the most systematic coverage available in Italian.