What does "Two-Sided Hypothesis" mean?

Definition of Two-Sided Hypothesis in the context of A/B testing (online controlled experiments).

What is a Two-Sided Hypothesis?

A two-sided hypothesis is an alternative hypothesis which is not bounded from above or from below, as opposed to a one-sided hypothesis which is always bounded from either above or below. In fact, a two-sided hypothesis is nothing more than the union of two one-sided hypotheses. For example, H1: δ < 0 ∪ δ > 0 (alt.: H1: θ∈(-∞,0) ∪ θ∈(0,+∞)) is a two-sided hypothesis. The corresponding null hypothesis would necessarily be a point hypothesis: H0: δ = 0 (alt.:H0: θ∈[0])).

A two-sided alternative hypothesis can be used when one wants to set his type I error against a very precise null hypothesis, for example that the effect is exactly zero (no smaller, no larger). A two-sided hypothesis will not be applicable to a superiority test which are most A/B tests performed, nor will it apply in the less-common non-inferiority test. As a statement it corresponds to the claim that the treatment will perform either better or worse than the control.

Despite its intuitive appeal, this is rarely the claim we want to defend via an online controlled experiment and the counter-position is rarely a strict "no effect" claim. Most of the time stakeholders will only approve a proposed change to a website, app or software if it is better than the existing state of affairs and will certainly not approve it if it is worse, therefore their position corresponds to a one-sided null hypothesis which begs the complimentary one-sided alternative. In fact, you will likely have trouble keeping your job as a conversion rate optimization specialist if you frequently allow tests to reach statistical significance for an effect in the negative direction (sequential testing with a futility boundary can help prevent that).

When a two-sided hypothesis is used the respective p-value should also be two-sided (or a two-tailed test as it is sometimes called). If a confidence interval is used to support a two-sided claim it should also be two-sided.

Articles on Two-Sided Hypothesis

Glossary Index by Letter


Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.