What does "Key Performance Indicator" mean?

Definition of Key Performance Indicator in the context of A/B testing (online controlled experiments).

What is a Key Performance Indicator?

Aliases: KPI

A Key Performance Indicator (KPI) is a metric which conveys important business information to relevant stakeholders. Being a metric means it has to be measurable, ideally in near real-time, and it also has to be actionable. "Key" is contextual and a metric which is important for some stakeholders may be much further down the list for others or not in their view altogether. In an A/B test "performance indicator" is usually translated to "metric", "variable" or "statistic".

A KPI can also be a composite, usually weighted metric based on several measurements. The issue in such cases is that a change in the KPI is less informative in itself and a break-down of the changes by individual measurement may be required.

Since a KPI is contextual, any metric can be a KPI given the proper context. When presenting to a design team important KPIs may be Average Click-Through Rate (CTR) on a certain Call-To-Action element (CTA). A head of IT will likely be more interested in a KPI like average page speed, average time to first paint, etc. An executive from the marketing department or a CEO will likely be most interested in metrics like average revenue per user (ARPU), average cost per lead, and so on.

A simple online controlled experiment will usually have just one parameter of interest, one random variable to work with: the primary KPI. More complex setups may have co-primary KPIs, as well as one or more secondary KPIs. This depends on the hypothesis one is looking to test and the resulting statistical design. When more than one key performance indicator is involved and the data from just one of them guides decisions about the outcome of the whole test a multiple testing situation arises which needs to be reflected in the statistical model to avoid mis-specification and faulty causal inference and/or estimation.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.