## What is Percent Change?

Aliases: *lift, relative change, relative difference*

Percent change, a.k.a. percent lift, relative change or relative difference, is the difference between two variables of interest relative to one of the variables. The percentage lift on some Key Performance Indicator (KPI) is often what is reported as the outcome of an A/B test, but often it is done using incorrect methods (see below).

The relative change is calculated as the absolute difference between two variables divided by one of them, usually that of the control group. To get a percent change one needs to multiply the relative change by 1000. Relative change is often written as δ_{Rel(B,A)} or δ_{Rel} = (μ_{2} - μ_{1})/μ_{1}, equivalently δ_{abs} = (p_{2} - p_{1})/p_{1} for proportions. For example, if the proportion in group B is 0.05 and the proportion in group A is 0.04, then δ_{Rel(B,A)} = (0.05 - 0.04) / 0.04 = 0.01 / 0.04 = 0.25. Multiply by 100 will give us 25% percent change.

Since most of the time in A/B testing the absolute differences are very small and often not very informative unless one also communicates the baseline, we prefer to report the relative difference (percent change, % lift) since calculating business effect is more straightforward when doing so. Since most traditional significance tests and confidence intervals are developed to address the question in terms of absolute difference, it is no wonder that many CRO practitioners get this wrong and fail to use a method that appropriately takes into account the added variance introduced by dividing by one of the variables.