What does "Type II Error" mean?

Definition of Type II Error in the context of A/B testing (online controlled experiments).

What is a Type II Error?

Aliases: false negative

A type II error a.k.a. an error of the second kind is committed when we fail to reject a false null hypothesis (erroneously accept the alternative hypothesis). For example, assuming the null hypothesis is that of no difference, even though there is in fact a difference of magnitude (μ) between the means of the Control Gropu and Test Group(s) we fail to observe a statistically significant difference between them after performing an online controlled experiment.

After an A/B Test is completed we have either committed a type II error, or we have not. The type II error rate is thus, inevitably, a characteristic of the testing procedure, not of the tested hypothesis. A properly designed and executed significance test offers conservative guarantees regarding the probability of committing a type II error. In fact, we know that with many different statistics we can devise an α-Uniformly Most Powerful Test so that we have optimal power given a certain sample size. However, the type II error guarantees only hold if the test actually performed conforms to the test as it was planned.

The type II error of a test is at odds with the type I error: increasing one leads to decreasing the other, and vice versa, assuming fixed variance, sample size and minimum effect of interest.

Articles on Type II Error

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.