What does "Percent Change" mean?

What is Percent Change?

Aliases: lift, relative change, relative difference

Percent change, a.k.a. percent lift, relative change or relative difference, is the difference between two variables of interest relative to one of the variables. The percentage lift on some Key Performance Indicator (KPI) is often what is reported as the outcome of an A/B test, but often it is done using incorrect methods (see below).

The relative change is calculated as the absolute difference between two variables divided by one of them, usually that of the control group. To get a percent change one needs to multiply the relative change by 1000. Relative change is often written as δ_Rel(B,A) or δ_Rel = (μ₂ - μ₁)/μ₁, equivalently δ_abs = (p₂ - p₁)/p₁ for proportions. For example, if the proportion in group B is 0.05 and the proportion in group A is 0.04, then δ_Rel(B,A) = (0.05 - 0.04) / 0.04 = 0.01 / 0.04 = 0.25. Multiply by 100 will give us 25% percent change.

Since most of the time in A/B testing the absolute differences are very small and often not very informative unless one also communicates the baseline, we prefer to report the relative difference (percent change, % lift) since calculating business effect is more straightforward when doing so. Since most traditional significance tests and confidence intervals are developed to address the question in terms of absolute difference, it is no wonder that many CRO practitioners get this wrong and fail to use a method that appropriately takes into account the added variance introduced by dividing by one of the variables.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Percent Change

Confidence Intervals & P-values for Percent Change / Relative Difference
blog.analytics-toolkit.com