## What is a Multivariate Test?

Aliases: *MVT, multivariate testing, A/B/n Test*

A multivariate test is one in which **more than one treatment variant is tested against a control group**. Usually the control is denoted "A" while the tested variants "B", "C", "D", etc.

Usually, such a test requires p-value adjustments in order to keep the nominal p-value in check due to the fact that introducing more and more variants and making a decision against the control based on just one significant outcome increases the probability of committing a type I error and thus alpha since.

Bonferroni derived the calculation that the overall α of comparing **m** variants versus a control by using a significance test is equal to **1 - (1 - α _{per comparison})^{m}** which is the probability that one of them will result in a statistically significant outcome. The simple Bonferroni correction would suggest testing each pair at level α/m to maintain the Family-Wise Error Rate (FWER) fixed at α, but this is a conservative adjustment when the comparisons are not independent.

The most-powerful method that maintains control of the FWER in such a scenario is the Dunnett's correction which makes use of the dependence between the comparisons to achieve greater power. Sample size computations and power analysis should be carried by taking the p-value correction into account.

An MV design test should not be confused with a factorial design test since the latter has an entirely different goal: isolating the effect of individual changes and can thus be an underpowered test with respect to higher-order interactions. An MVT can still be interpreted in terms of contribution of the different elements by combining all users/sessions/etc. exposed to a factor across the test groups and comparing their performance versus those who were not (again across all test groups).

## Articles on Multivariate Test

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.