What does "Primary KPI" mean?

What is a Primary KPI?

The primary KPI (key performance indicator) is the primary metric for which an experiment is planned and by which its outcome will be judged. For example, in an A/B test in conversion rate optimization one often has some kind of conversion rate as a primary KPI, although other metrics such as average revenue per user is also often employed. When performing sequential testing the primary KPI is the one used to make the decision to stop or continue the test.

A primary KPI has to be chosen based on the primary question of interest the test it is supposed: the hypothesis/claim it is testing. Furthermore, when choosing it one needs to keep in mind that the closer it is to the business bottom-line, the more informative it is, but also it usually takes longer to test due to certain statistical realities. However, the trade-off is often preferred, especially when the assumption that an effect on a metric further up the funnel will maintain its effect further down.

In many online controlled experiments a co-primary KPI is also selected, e.g. in a test that aims to improve the overall purchase rate the primary metric will be the transaction rate per user with a superiority test while a co-primary metric can be average revenue per user tested by a non-inferiority test.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.