What does "A/A Test" mean?

Definition of A/A Test in the context of A/B testing (online controlled experiments).

What is an A/A Test?

An A/A test is a particular form of an A/B test in which there is no difference between the experience of the test group and the control group. An A/A test is thus expected to produce no statistically significant results given a low significance threshold is required to declare an outcome significant. Furthermore, it is expected that the cumulative performance of the groups gets closer and closer to each other with time, demonstrating the expected asymptotic consistency of the mean statistic.

For these reasons an A/A test with a high significance threshold is useful in assuring that there are no issues with the testing setup such as improper randomization, systemic bias due to technical issues (e.g. "flicker", slower loading speed when choosing a treatment arm, etc.) and so on. One needs to be aware of the cost of doing so (opportunity costs), especially if one decides to wait for an A/A test to complete before running an actual A/B test. It is better to run them in parallel, as concurrent tests, when possible.

An A/A test can also be a useful teaching device with which one can illustrate the need to use statistical estimates of uncertainty when reporting values as well as demonstrate more complex issues such as peeking with intent to stop. On a side-note, running an A/A/A/A... test after one is sure of their platform due to running many A/A tests can illustrate the need for control of the Family-Wise Error Rate.

Related A/B Testing terms

A/B Test

Articles on A/A Test

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.