What does "Significance Threshold" mean?

What is a Significance Threshold?

The significance threshold is chosen during the planning of an A/B test and it corresponds to the probability of committing a type I error (registering a false positive) which is deemed acceptable under the specific circumstances of the test in question. The threshold is used to compute the sample size needed for a uniformly most powerful test at that threshold and specified minimum effect of interest and statistical power against a composite hypothesis with a lower bound at the MEI.

After the test is completed, the observed p-value is compared to the threshold and if it is lower the null hypothesis is rejected.

The significance threshold is often set to 0.05 (equivalent to 5% confidence level) but when choosing the significance threshold for a particular test one should ideally consider the particular risks and rewards associated with the test at hand. A test for a major decisions which has wide-ranging consequences and is hard to reverse might require a very high evidential threshold, say 0.001. On the other hand, a different test in which the decision has limited scope and is easy to reverse if necessary can be planned with a much higher threshold (lower evidential input) of 0.1. Sample size and test duration considerations also enter into account.

A particular value of the significance threshold is usually denoted in formulas as c_(α) where α is alpha and the "c" comes from another term which is often used in statistical literature: critical region.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Articles on Significance Threshold

Statistical Significance in A/B Testing – a Complete Guide
blog.analytics-toolkit.com

P-value Definition and Interpretation
www.gigacalculator.com