What does "A/A Test" mean?

What is an A/A Test?

An A/A test is a particular form of an A/B test in which there is no difference between the experience of the test group and the control group. An A/A test is thus expected to produce no statistically significant results given a low significance threshold is required to declare an outcome significant. Furthermore, it is expected that the cumulative performance of the groups gets closer and closer to each other with time, demonstrating the expected asymptotic consistency of the mean statistic.

For these reasons an A/A test with a high significance threshold is useful in assuring that there are no issues with the testing setup such as improper randomization, systemic bias due to technical issues (e.g. "flicker", slower loading speed when choosing a treatment arm, etc.) and so on. One needs to be aware of the cost of doing so (opportunity costs), especially if one decides to wait for an A/A test to complete before running an actual A/B test. It is better to run them in parallel, as concurrent tests, when possible.

An A/A test can also be a useful teaching device with which one can illustrate the need to use statistical estimates of uncertainty when reporting values as well as demonstrate more complex issues such as peeking with intent to stop. On a side-note, running an A/A/A/A... test after one is sure of their platform due to running many A/A tests can illustrate the need for control of the Family-Wise Error Rate.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on A/A Test

Should you do A/A, A/A/B or A/A/B/B tests in CRO?
blog.analytics-toolkit.com