What does "Hypothesis Testing" mean?

What is Hypothesis Testing?

Hypothesis testing involves the act of trying to disprove a hypothesis (H) by gathering data (evidence, e) which has high probability of disproving H if H is false. In an A/B test this is usually done via performing a Null Hypothesis Statistical Test that has desired properties such as known error rates, unbiasedness and so on. A Significance Test is usually employed for the purpose.

Due to the impossibility to "confirm" (in the strict sense or probabilistically) a hypothesis (H->e, e ∴ H does not follow), one sets out to try and disprove a hypothesis using the logic H->e, not-e ∴ not-H which is valid. The process generally starts with the translation of a substantive claim of interest into a statistical model than can be used to estimate the probability of observing one outcome or another under the assumption that H is true. A certain level of evidence (significance threshold or confidence level) has to be agreed on as being stringent enough for the task at hand.

After the online controlled experiment it designed and executed the data is analyzed and a decision is reached based on the agreed upon parameters. Estimates for the parameters of interest can also be provided which is why many people do not differentiate strictly between hypothesis testing and estimation.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.