What does "Return on Investment" mean?

Definition of Return on Investment in the context of A/B testing (online controlled experiments).

What is Return on Investment?

Alias: ROI

Return on Investment (ROI) is a simple ratio between the resources invested in a given project and the return in terms of business revenue it generated. It can be calculated using the formula ROI = Revenue / Investment. It is often communicated as a percentage, which requires the result from the previous formula to be multiplied by 100%. For example, if the investment is $2,000 and the return is $3,000 we have 3000/2000 * 100% = 150% ROI.

In A/B testing we can talk about the return on investment of a particular A/B test, a series of A/B tests or even a whole experimentation program encompassing dozens or hundreds of online experiments over years.

If we are analyzing the return on investment of a particular test, we need to account for its total cost and its total rewards, accounting for the risk incurred (both in terms of risk during the test, as well as the risk from making the wrong decision: type I error or type II error). During the test planning phase one can perform a risk-reward analysis to determine the significance threshold and test duration that would result in an optimal ROI. In this sense one can come up with a statistical design that is ROI-optimal.

Analyzing the ROI of an entire testing program is more difficult due to the amount of accounting data that needs to be gathered, as well as the intricacies of the meta-analysis required to aggregate the returns from the A/B tests.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.