What does "Risk-Reward Analysis" mean?

What is Risk-Reward Analysis?

Risk-Reward analysis is the practice of weighing the expected risks and rewards from an A/B test and arriving at an optimal statistical design for it based on the trade-offs involved. The outcome of a risk-reward analysis is an optimal significance threshold and test duration/sample size. The latter can also be expressed in terms of optimal statistical power level for a fixed minimum effect of interest or the MEI at XX% power (customarily 80%).

A risk-reward analysis is carried out after it is decided what the test will involve but before actually starting it. The result of such an analysis may also produce information like the risk/reward ratio, the marginal improvement in the ratio versus implementing without testing, as well as key points of interest such as the break-even point, the maximum probability-adjusted gains and maximum probability-adjusted loss. If the risk/reward ratio is less than 1.00 the analysis suggests that even an optimal online controlled experiment is worse than implementing the proposed change directly - a situation which can arise from different conditions.

The inputs for a risk-reward analysis are basic information about the daily/weekly/monthly users a website/app/software sees, the historical value of the parameter of interest (the primary KPI), the revenue generated, as well the projected cost and reward, usually broken down by major components.

The analysis also involves the stipulation of an expected probability distribution for the key performance indicator which is usually informed by prior experience with similar tests as well as conjectures based on other experience. Unlike Bayesian inference it is not used to produce a posterior distribution since a risk-reward analysis is limited to the choice of test parameters before a test commences and does not make use of any test data. This distribution is, however, of significant importance as it serves as the basis for all probability-adjusted factors.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Risk-Reward Analysis

Risk vs. Reward in A/B Tests: A/B testing as Risk Management
blog.analytics-toolkit.com

Costs and Benefits of A/B Testing: A Comprehensive Guide
blog.analytics-toolkit.com

Inherent costs of A/B testing: limited risk results in limited gains
blog.analytics-toolkit.com