What does "Reward" mean?

What is Reward?

In A/B testing a reward is the positive value return one makes after successfully identifying and implementing an improvement to a website, app or other software. In particular, we can differentiate between two types of gains: fixed gains which can be due to cost-saving and gains that depend on the actual improvement we have achieved (%lift). In a risk-reward analysis the former are called fixed gains and the latter: probability-adjusted gains.

By performing an A/B test we always generate less revenue than we would otherwise if the true reward is positive. In particular, a business will generate less revenue due to delaying the release of a truly better experience to 100% of the users. Further risk will be incurred due to the limited sensitivity statistical power of all A/B tests towards (relatively) small true effects. The combined effect is that while an online controlled experiment reduces the business risk associated with a particular decision, it also limits the rewards the decision would generate if it were indeed profitable.

Reward takes a slightly different meaning during a risk-reward analysis where the reward function is calculated based on a prior distribution. Based on that prior we can estimated the probability-adjusted gains a business can achieve both during and after an A/B test. Such a calculation is entirely counterfactual as it is performed pre-data.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Reward

Risk vs. Reward in A/B Tests: A/B testing as Risk Management
blog.analytics-toolkit.com

Costs and Benefits of A/B Testing: A Comprehensive Guide
blog.analytics-toolkit.com

Inherent costs of A/B testing: limited risk results in limited gains
blog.analytics-toolkit.com