What does "Risk" mean?
Definition of Risk in the context of A/B testing (online controlled experiments).
What is Risk?
In A/B testing risk is the probability associated with a negative event, and event with negative expected value. In particular, we are concerned with the risk of arriving at a wrong conclusion. We can measure this risk by performing an A/B test where the risk of committing a type I error or a type II error are controlled via the error probability guarantees of the procedure we use (usually a significance test). The two types of risk are denoted alpha and beta.
A different kind of framing risk is through an uncertainty related to an estimate. A common method to convey that is the construction of confidence intervals. Yet another approach to assessing risk is to calculate the severity related to any claim about a parameter of interest.
Risk takes a slightly different meaning during a risk-reward analysis where the risk function is calculated based on a prior distribution. Based on that prior we can estimated the risk-adjusted loses we can incur both during and after an A/B test. The difference between a pre-test risk-adjusted loss and the type I error is that the latter is a conservative bound based on the most extreme value of the null hypothesis and does not consider any information external to the test while the former incorporates different types of information and also incorporates loses due to type II errors. Also, the risk-adjusted loss is counterfactual and based entirely on a loss function while the type I error is entirely factual.
It is important to understand that limiting risk always comes at the cost of limited gains (reward).
Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.
Articles on Risk
Risk vs. Reward in A/B Tests: A/B testing as Risk Management
blog.analytics-toolkit.com
Costs and Benefits of A/B Testing: A Comprehensive Guide
blog.analytics-toolkit.com
Inherent costs of A/B testing: limited risk results in limited gains
blog.analytics-toolkit.com
Related A/B Testing terms
Significance ThresholdStatistical PowerCostRewardRisk-Reward AnalysisReturn on InvestmentMinimum Effect of InterestSee this in action
A/B Test Planner
Statistical Methods in Online A/B Testing
Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.
Learn moreGlossary index by letter
Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.