What does "Adaptive Sequential Design" mean?

What is an Adaptive Sequential Design?

Alias: ASD

An adaptive sequential design as a set of approaches to sequential testing in which the statistical design of the A/B test is not fixed for its entire duration but instead changes (adopts) based on the data currently gathered and projections from it. Adaptive sequential designs are an extension of the group-sequential design (GSD) approach and so they share a lot in common such as the use of error-spending functions. On top of what one can do in a GSD, in an ASD one can change the number of test groups, the proportion of users allocated in each group, as well as the total sample size via sample-size re-estimation.

For example, a group may be stopped entirely due to poor performance, or, more rarely, a new group may be added along the way to test something that was informed by the data in the existing groups. If a certain group is performing worse than other groups fewer new users would be allocated to it and vice versa. If the observed effect size is different than the one the test was powered for adjustments to the sample size can be made based on the ratio between planned minimum effect of interest and observed effect size or based on conditional statistical power.

While an adaptive sequential design allows for great flexibility in conducting a test it also comes at the cost of significant increase in the computational difficulty in assessing the results and calculating a p-value and a confidence interval with good properties. Communication to stakeholders may also be impeded as result.

Most importantly, an ASD is never more efficient than a classic GSD. It has been proven in several works that for any ASD there exists at least an equally efficient group-sequential design and oftentimes there exists a mildly or moderately more efficient one. Thus, there is no argument for the use of an adaptive sequential design based on its efficiency.

It should be stressed that the flexibility allowed by an adaptive sequential experiment: adding or dropping groups, changing the allocation during the test as well as sample size re-estimation, if desirable, cannot be done haphazardly and should instead follow proper methods for making the changes and apply the necessary p-value adjustements and corrections of other estimates as necessary.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.