What does "Cost" mean?

What is Cost?

In A/B testing the cost related to an A/B test are any business expenses made with relation to a test, including estimated risk. It is the sum total of the fixed-cost investment plus risk-adjusted costs.

Examples for fixed costs are: price of testing software (third-party or internal), compensation for conversion rate optimization specialists, development and designer time dedicated to a test, as well as time from other relevant parties, cost of tracking software and its management, as well as any temporary costs specific to the test. Risk-adjusted costs include the risk of exposing users to an inferior experience, the loses incurred due to the missed opportunity to capitalize on a superior experience as early as possible, as well as the risk of failing to reject a truly inferior current experience in favor of a superior one (type II error).

In relation to time costs can be divided into: planning costs, execution costs (during the duration of the A/B test), and exploitation costs (the risk of implementing an inferior experience, type I error).

In a pre-data risk-reward analysis risk cost is calculated based on a prior distribution. Based on that prior we can estimated the risk-adjusted loses we can incur both during and after an A/B test. The difference between a pre-test risk-adjusted loss and the type I error is that the latter is a conservative bound based on the most extreme value of the null hypothesis and does not consider any information external to the test while the former incorporates different types of information and also incorporates loses due to type II errors. Also, the risk-adjusted loss is counterfactual and based entirely on a loss function while the type I error is entirely factual.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Cost

Costs and Benefits of A/B Testing: A Comprehensive Guide
blog.analytics-toolkit.com

Inherent costs of A/B testing: limited risk results in limited gains
blog.analytics-toolkit.com