What does "Test Duration" mean?

Definition of Test Duration in the context of A/B testing (online controlled experiments).

What is Test Duration?

The duration of an A/B test is measured in the number of hours, days, weeks or months during which it is conducted. Most tests would be planned for and measured in weeks as this provides better generalizability of the results, but some tests can produce a representative sample in mere days, assuming a high enough daily user count and lack of within-week variability.

The duration of an online controlled experiment is determined by the required sample size (estimated through a risk-reward analysis which involves a power analysis) and the amount of daily/weekly users eligible to participate in the test. Naturally this involves projections for the number of users the site will attract in a future period and seasonality should be taken into account.

In certain cases there are external considerations that set deadlines on the release of the result of an A/B test so the maximum duration may be set on that basis with the actual duration being restricted on being less than or equal to the maximum. In this case conducting a risk-reward analysis can still drive the decision to test for a shorter amount of time. A power analysis should be conducted even in the case of a restricted maximum duration even if only to inform about the minimum detectable effects at certain points of the power function.

Related A/B Testing terms

Sample Size

Sequential Testing

Cookie Churn

Articles on Test Duration


Glossary Index by Letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.