Statistical Parametric and Non-Parametric Tests

Have you ever run an A/B test and wondered whether you should use a t-test or a Wilcoxon test? The choice matters: using the wrong test can cause you to miss a significant result — or, worse, make you think a result is significant when it’s not.

Statistical tests fall into two broad families: parametric and non-parametric. The fundamental difference? Parametric tests assume the data follow a known distribution (usually the normal); non-parametric tests make no such assumption. This is not a technical detail — it determines which tests you can use, how powerful they are, and how much you can trust the results.

In this article we’ll look at the differences, when to use each, and how the choice impacts the analysis of your site’s data.

What we’ll cover

• Parametric tests: the power of normality
• Non-parametric tests: flexibility and robustness
• Comparison table
• Try it yourself
• Choosing the right test

---

Parametric tests: the power of normality

Many parametric tests assume the data are an approximately normally distributed.

The typical conditions for using them are:

• Continuous or interval/ratio data.
• Approximately normal distribution of the data (or of sample means, thanks to the central limit theorem for large samples).
• Homogeneity of variances (homoscedasticity) across groups.
• A sufficiently large sample, when the model assumptions are reasonably met.

The main advantage of parametric tests is higher statistical power: for the same effect size and sample size, a parametric test is more likely to detect a real difference. This is because they leverage the information about the data’s distribution — provided the assumption holds.

Examples of parametric tests we have covered include the Z-test, Student’s t-test, ANOVA (Analysis of Variance), the Pearson correlation coefficient r, and linear regression.

---

Non-parametric tests: flexibility and robustness

Non-parametric tests (also called distribution-free tests) do not require assuming a particular distribution of the data nor estimating the parameters of such a distribution. They often work on ranks (the ordering of values) rather than on the values themselves.

They fall into two broad categories:

• Goodness-of-fit tests: compare observed frequencies with those expected under a hypothesis. The prime example is the chi-square test.
• Non-parametric alternatives to parametric tests: tests that answer the same question without assuming normality. For example, the Wilcoxon test is the alternative to the t-test for two samples, and Spearman’s rho is the non-parametric alternative to Pearson’s correlation.

Other examples include the Mann-Whitney U test (alternative to the t-test for two independent samples), the Kruskal-Wallis test (alternative to ANOVA), and Kendall’s tau (alternative to Spearman for ties).

The price of this versatility? Non-parametric tests generally have lower statistical power when the assumptions of parametric tests are met. They have lower efficiency: to achieve the same power, you need larger samples.

---

Comparison table

When to use each? Here is a summary:

Feature	Parametric tests	Non-parametric tests
Assumed distribution	Normal (approx.)	None
Data type	Continuous, interval/ratio	Any (including ordinal, ranks)
Statistical power	Higher (if assumptions hold)	Lower (same sample size)
Robustness	Sensitive to outliers, non-normality	Robust to outliers and skewness
Minimum sample	Depends on model assumptions	Works even with small n
Output	Parameter estimates + p-value	Rank-based test statistic
Examples	t-test, ANOVA, Pearson, Z-test	Wilcoxon, Mann-Whitney, Kruskal-Wallis, Spearman

---

Try it yourself

An ecommerce site is testing two versions of a product page (A and B). Session times in seconds for 10 visitors each are:

version_A <- c(45, 52, 48, 61, 55, 49, 53, 47, 58, 51)
version_B <- c(42, 65, 44, 70, 50, 68, 55, 71, 49, 63)

Let’s run both a t-test (parametric) and a Wilcoxon test (non-parametric) to see if the difference is significant:

# Parametric test (assumes normality)
t.test(version_A, version_B)

# Non-parametric test (rank-based)
wilcox.test(version_A, version_B)

Questions:

1. Do the two tests give the same verdict (p < 0.05)?

2. Which test makes more sense given only 10 data points per group?

3. How would the choice change with 100 visitors per version?

Answers:

1. With this particular data, the t-test gives p ≈ 0.12 (not significant), while Wilcoxon gives p ≈ 0.16 (not significant). They agree — but that’s not always the case.

2. With n = 10 it’s hard to verify normality. Wilcoxon is the safer choice: it makes no distributional assumptions and works well with small samples.

3. With n = 100, the central limit theorem guarantees the sample mean is approximately normal even if the original data aren’t. The t-test becomes more justified — and more powerful.

---

Choosing the right test

The choice between a parametric and a non-parametric test does not depend on which is "best" in absolute terms, but on the nature of the data and the assumptions we are reasonably willing to make.

The rule of thumb is:

• If the data are normal (or the sample is large) and the variances are homogeneous → parametric test (more powerful).
• If the data are not normal, the sample is small, or there are outliers → non-parametric test (more robust).
• When in doubt, run both: if they agree, the answer is robust. If they disagree, investigate why.

---

Further Reading

For a comprehensive treatment of the choice between parametric and non-parametric tests, with the applicability conditions of each, Statistica by Newbold, Carlson and Thorne offers the most systematic coverage available in Italian.