What does "Multivariate Test" mean?

What is a Multivariate Test?

Aliases: MVT, multivariate testing, A/B/n Test

A multivariate test is one in which more than one treatment variant is tested against a control group. Usually the control is denoted "A" while the tested variants "B", "C", "D", etc.

Usually, such a test requires p-value adjustments in order to keep the nominal p-value in check due to the fact that introducing more and more variants and making a decision against the control based on just one significant outcome increases the probability of committing a type I error and thus alpha since.

Bonferroni derived the calculation that the overall α of comparing m variants versus a control by using a significance test is equal to 1 - (1 - α_{per comparison})^m which is the probability that one of them will result in a statistically significant outcome. The simple Bonferroni correction would suggest testing each pair at level α/m to maintain the Family-Wise Error Rate (FWER) fixed at α, but this is a conservative adjustment when the comparisons are not independent.

The most-powerful method that maintains control of the FWER in such a scenario is the Dunnett’s correction which makes use of the dependence between the comparisons to achieve greater power. Sample size computations and power analysis should be carried by taking the p-value correction into account.

An MV design test should not be confused with a factorial design test since the latter has an entirely different goal: isolating the effect of individual changes and can thus be an underpowered test with respect to higher-order interactions. An MVT can still be interpreted in terms of contribution of the different elements by combining all users/sessions/etc. exposed to a factor across the test groups and comparing their performance versus those who were not (again across all test groups).

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Multivariate Test

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests
blog.analytics-toolkit.com