What does "Online Controlled Experiment" mean?

What is an Online Controlled Experiment?

Alias: OCE

The term "online controlled experiment" is a generic term that encompasses all kinds of online experiments set up for learning from data. It is the golden standard for drawing objective causal inferences in online marketing, user experience and conversion rate optimization. While the term A/B test is often used instead, OCE is the technically-correct one to use if one is speaking for something more complex than a simple test of one variant versus a control.

An online controlled experiment, by definition, employs a control group and one or more treatment groups among which users are assigned randomly. Ranodmization allows modelling the effect of all confounding factors as random variables and thus allows one to estimate the effect of the treatment in a statistically rigorous way.

From the point of an online business we can distinguish between two main goals of using online controlled experiments: to control the business risk associated with a making a certain decision and to estimate the magnitude and direction of an effect. The first part is achieved through hypothesis testing while the second is accomplished through estimation. Often a practitioner will want both. Usually the former is the primary goal and the latter the secondary.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Online Controlled Experiment

A/B Testing Statistics - A Concise Guide
blog.analytics-toolkit.com