What does "Factorial Design" mean?

What is a Factorial Design?

A factorial design is a type of online controlled experiment in which the outcome of interest is the contribution of individual factors to a variable of interest (often a KPI) and not higher-order interactions between these factors. A factorial design requires a smaller sample size compared to an equivalent multivariate test if it is not powered to detect higher-order interactions and requires the same sample size if powered for them (see Power Analysis).

In most cases CRO practitioners are interested not only about the effect of certain factors, but also of the combined effect of multiple factors changed together. When that is the case, a properly designed MVT can provide insights both on the performance of the combined changes as well as the contribution of each individual factor. Given that MVTs are much easier to set up and analyze properly they are often the preferred option.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Factorial Design

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests
blog.analytics-toolkit.com