What does "Sidak Correction" mean?

Definition of Sidak Correction in the context of A/B testing (online controlled experiments).

What is the Sidak Correction?

The Šidák correction is a method for controlling the Family-Wise Error Rate (FWER) in the strong sense and it is only slightly less conservative than the most conservative Bonferroni correction. This test produces a familywise Type I error rate of exactly α when the tests are independent from each other and all null hypotheses are true. For positively-correlated tests the procedure is conservative and it is liberal for negatively-correlated tests.

In practice the Šidák correction a good method when one compares two or more groups across a range of metrics, for example comparing bounce rate, conversion rate, average revenue per user and average site speed and considering each of the comparisons enough to make a call on the test. The correction is usually implemented by using the Šidák Step-Down procedure.

Related A/B Testing terms

Multiple Testing

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.