What does "Test Duration" mean?

What is Test Duration?

The duration of an A/B test is measured in the number of hours, days, weeks or months during which it is conducted. Most tests would be planned for and measured in weeks as this provides better generalizability of the results, but some tests can produce a representative sample in mere days, assuming a high enough daily user count and lack of within-week variability.

The duration of an online controlled experiment is determined by the required sample size (estimated through a risk-reward analysis which involves a power analysis) and the amount of daily/weekly users eligible to participate in the test. Naturally this involves projections for the number of users the site will attract in a future period and seasonality should be taken into account.

In certain cases there are external considerations that set deadlines on the release of the result of an A/B test so the maximum duration may be set on that basis with the actual duration being restricted on being less than or equal to the maximum. In this case conducting a risk-reward analysis can still drive the decision to test for a shorter amount of time. A power analysis should be conducted even in the case of a restricted maximum duration even if only to inform about the minimum detectable effects at certain points of the power function.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Test Duration

20-80% Faster A/B Tests? Is it real?
blog.analytics-toolkit.com