What does "Overall Evaluation Criterion" mean?

What is an Overall Evaluation Criterion?

Alias: OEC

An Overall Evaluation Criterion (OEC) is a (usually composite) quantitative measure of the experiment’s objective. Other names include Response or Dependent Variable, Outcome Variable, Evaluation metric, Performance metric. Most often one speaks of an overall evaluation criterion when a single key performance indicator is deemed insufficient for the evaluation of the outcome of an online controlled experiment and separate evaluation of metrics is not desirable for some reason. In such a case a weighted combination KPIs as a primary KPI is deemed desirable and is recommended as long as there is agreement on the relative weights.

According to R.Kohavi et al. ^[1] "a single metric forces trade-offs to be made once for multiple experiments and aligns the organization behind a clear objective. A good OEC should not be short-term focused (e.g., clicks); to the contrary, it should include factors that predict long-term goals, such as predicted lifetime value and repeat visits." So, ideally, an OEC is based on metrics in a short-term experiment that are good predictors of long-term value.

An example provided by Kohavi et. al. is "units purchased, revenue, profit, expected lifetime value, or some weighted combination of these", but a conversion rate optimization practitioner should make the decision on a suitable overall evaluation criterion based on the particular business objectives, measurement capabilities and established knowledge from prior experimental and observational data.

References:
[1] Kohavi R. et al. (2009) "Controlled experiments on the web - survey and practical guide"

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.