How-to articles

Glossary of A/B testing terms

Alphabetical index of 219 terms and abbreviations in A/B testing / online controlled experiments in Conversion Rate Optimization (CRO).

Jump to terms by letter

A B C D E F G H I K L M N O P R S T U V Z

A

B

Bayesian Inference Bernoulli Metric Beta-Spending Beta-spending Function Bias Binomial Metric Bonferroni Correction Bounce Rate beta

C

D

Delta Dependent Variable Distribution Dunnett's Correction

E

Effect Size Efficacy Boundary Efficient Estimator Error Probability Error Rate Error-Spending Function Estimation Estimator External Validity

F

FWER Factorial Design False Discovery Rate False Negative False Negative Rate False Positive Family-Wise Error Rate Frequentist Inference Futility Boundary

G

GSD GST Gaussian Distribution Generalizability Group-sequential Design

H

Holdout Group Hypothesis Hypothesis Testing

I

ID Identically Distributed Independent Observation Independent Variable Interaction Effects Intervention

K

KPI Key Performance Indicator

L

Learning Effect Lift Likelihood Likelihood Function

M

MDE MEI MLE MSE MVT Maximum Likelihood Estimate Mean Mean Square Error Minimum Detectable Effect Minimum Effect of Interest Modus Tollens Mu Multi-Armed Bandit Multiple Comparisons Multiple Testing Multivariate Test Multivariate Testing

N

NHST Nominal Significance Nominal p-value Non-Binomial Metric Non-Inferiority Margin Non-Inferiority Test Non-Parametric Test Normal Distribution Novelty Effect Null Hypothesis Null Hypothesis Statistical Test

O

OCE OEC Observed Significance One-Sided Hypothesis One-Tailed Test One-sided Test Online Controlled Experiment Optional Stopping Optional Stopping Overall Evaluation Criterion Overpowered Test

P

Parametric Test Peeking Percent Change Point Hypothesis Power Analysis Power Function Primary KPI Proportion Purchase Rate p-value p-value Adjustment p-value Distribution

R

ROI RPU Randomization Randomizer Rate Rejection Region Relative Change Relative Difference Representative Sample Representativeness Return on Investment Revenue per User Reward Risk Risk-Reward Analysis

S

T

T-test Test Duration Test Group Test Of Significance Theta Transaction Rate Treatment Two-Sided Hypothesis Two-Tailed Test Two-sided Test Type I Error Type II Error t Score

U

UMP Unbiased Unbiased Estimator Underpowered Test Uniformly Most Powerful Test

V

Variance Variant

Z

Glossary index by letter

A B C D E F G H I K L M N O P R S T U V Z

If you have a suggestion for an A/B testing term we should add to this glossary do not hesitate to contact us.