What does "Cost" mean?

Definition of Cost in the context of A/B testing (online controlled experiments).

What is Cost?

In A/B testing the cost related to an A/B test are any business expenses made with relation to a test, including estimated risk. It is the sum total of the fixed-cost investment plus risk-adjusted costs.

Examples for fixed costs are: price of testing software (third-party or internal), compensation for conversion rate optimization specialists, development and designer time dedicated to a test, as well as time from other relevant parties, cost of tracking software and its management, as well as any temporary costs specific to the test. Risk-adjusted costs include the risk of exposing users to an inferior experience, the loses incurred due to the missed opportunity to capitalize on a superior experience as early as possible, as well as the risk of failing to reject a truly inferior current experience in favor of a superior one (type II error).

In relation to time costs can be divided into: planning costs, execution costs (during the duration of the A/B test), and exploitation costs (the risk of implementing an inferior experience, type I error).

In a pre-data risk-reward analysis risk cost is calculated based on a prior distribution. Based on that prior we can estimated the risk-adjusted loses we can incur both during and after an A/B test. The difference between a pre-test risk-adjusted loss and the type I error is that the latter is a conservative bound based on the most extreme value of the null hypothesis and does not consider any information external to the test while the former incorporates different types of information and also incorporates loses due to type II errors. Also, the risk-adjusted loss is counterfactual and based entirely on a loss function while the type I error is entirely factual.

Articles on Cost

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.