What does "Concurrent Tests" mean?

What is Concurrent Tests?

The term concurrent tests describes a situation in which multiple online controlled experiments are running on the same website, app or software. They might be present on the same screen or page or might affect separate experiences.

The main concern in such cases are so-called interaction effects: interference between the test variants which causes the wrong winner to be selected at the end of a set of A/B tests running at the same time (with a certain overlap). While it is a valid concern, it is likely less common than most people think, since we will get the result of one of two concurrent A/B tests wrong only if there is a stronger and opposite in sign interaction in two of the four possible combination in such an A/B testing scenario.

Obviously, the issue gets more prominent when one runs dozens or even hundreds of tests at the same time at which point routine measures should be taken to ensure that there are no interaction effects of the above kind. This is mostly a concern for very large software companies and retailers.

One of the commonly proposed solutions is running tests in isolated lanes (testing silos) but instead of improving the situation it only worsens it as it leads, inevitably, to releasing untested combinations of user experiences for which no interaction effects can be estimated as the data is simply non-existent due to the isolation.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Concurrent Tests

Running Multiple A/B Tests at The Same Time: Do’s and Don’ts
blog.analytics-toolkit.com