What does "Secondary KPI" mean?

What is a Secondary KPI?

In an online controlled experiment one is often interested in measuring effects secondary to the main hypothesis under examination. For example, if in an A/B test one is looking at the conversion rate of a funnel, one will be happy to have it as a primary KPI and decide to stop or continue the test in a sequential testing scenario.

What if one is also interested in measuring the average page load speed? In the above A/B test it would not make sense to stop the test if it goes down a bit or if it is statistically significantly improved. However, it would make sense to keep track of it as a seconadary KPI (key performance indicator): one that needs to be monitored and can inform some decisions based on the data, but is not of primary interest during the present experiment.

So, a secondary KPI is an effect we would like to measure, but significant effects on which by itself would not constitute sufficient reason to terminate the test and make a decision in either direction.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.