What does "Confidence Level" mean?

What is a Confidence Level?

Confidence level is a measure of the coverage probability of a confidence interval, where "coverage probability" refers to the frequency with which a random interval build using this procedure contains a true (unknown) parameter (usually denoted θ*). It is a characteristic of the statistical procedure generating the interval. For example, if a confidence interval is constructed with a confidence level of 99% it means that 99% of such confidence intervals would contain the true parameter value.

After observing data x₀ and constructing a particular confidence interval we can accept (or reject) different hypothesized values of the parameter can be accepted according to whether they are contained within (or outside) the interval. One can construct confidence intervals at different confidence levels to see which values are rejected at which level of confidence.

The above, however, should not be confused with the interpretation that any particular value inside the interval is well-supported (supported at confidence level XX%, whatever that means): in fact each individual value within an interval has very little support on its own. One is not allowed to discriminate between values in the interval in such a way.

Confidence intervals can be one-sided or two-sided depending on whether one is looking to get a confidence limit with which to compare a hypothesized value (is it higher or lower than it) or if one is looking to simply get a range of values and has no intent of comparing in a specific direction. In the latter case one abstains from making claims about values that fall above or below an interval and talks talks about values that are either inside or outside.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.