Glossary of A/B testing terms

Alphabetical index of 213 terms and abbreviations in A/B testing / online controlled experiments in Conversion Rate Optimization (CRO).

Jump to terms by letter


ABCDEFGHIKLMNOPRSTUVZ


A

A/A Test

A/B Test

A/B/n Test

AGILE A/B Test

AOV

ARPU

ASD

Absolute Change

Absolute Difference

Actual Significance

Actual p-value

Adaptive Sequential Design

Alpha-Spending

Alpha-spending Function

Alternative Hypothesis

Argument from Coincidence

Average

Average Order Value

Average Revenue Per User

Average Sample Size

alpha


B

Bayesian Inference

Bernoulli Metric

Beta-Spending

Beta-spending Function

Bias

Binomial Metric

Bonferroni Correction

Bounce Rate

beta


C

CI

CR

CRO

Carry-Over Effect

Causal Inference

Chi Squared Test

Churn Rate

Composite Hypothesis

Concurrent Tests

Confidence Bound

Confidence Interval

Confidence Level

Confidence Limit

Confidence Threshold

Consistent Estimator

Continuous Metric

Control Group

Conversion Rate

Conversion Rate Optimization

Cookie Churn

Cost

Critical Region


D

Delta

Dependent Variable

Distribution

Dunnett's Correction


E

Effect Size

Efficacy Boundary

Efficient Estimator

Error Probability

Error Rate

Error-Spending Function

Estimation

Estimator

External Validity


F

FWER

Factorial Design

False Discovery Rate

False Negative

False Negative Rate

False Positive

Family-Wise Error Rate

Frequentist Inference

Futility Boundary


G

GSD

GST

Gaussian Distribution

Generalizability

Group-sequential Design


H

Holdout Group

Hypothesis

Hypothesis Testing


I

ID

Identically Distributed

Independent Variable

Interaction Effects

Intervention


K

KPI

Key Performance Indicator


L

Learning Effect

Lift

Likelihood

Likelihood Function


M

MDE

MEI

MLE

MSE

MVT

Maximum Likelihood Estimate

Mean

Mean Square Error

Minimum Detectable Effect

Minimum Effect of Interest

Modus Tollens

Mu

Multi-Armed Bandit

Multiple Comparisons

Multiple Testing

Multivariate Test

Multivariate Testing


N

NHST

Nominal Significance

Nominal p-value

Non-Binomial Metric

Non-Inferiority Margin

Non-Inferiority Test

Non-Parametric Test

Normal Distribution

Novelty Effect

Null Hypothesis

Null Hypothesis Statistical Test


O

OCE

OEC

Observed Significance

One-Sided Hypothesis

One-Tailed Test

One-sided Test

Online Controlled Experiment

Optional Stopping

Optional Stopping

Overall Evaluation Criterion

Overpowered Test


P

Parametric Test

Peeking

Percent Change

Point Hypothesis

Power Analysis

Power Function

Primary KPI

Proportion

Purchase Rate

p-value

p-value Adjustment

p-value Distribution


R

ROI

RPU

Randomization

Randomizer

Rate

Rejection Region

Relative Change

Relative Difference

Representative Sample

Representativeness

Return on Investment

Revenue per User

Reward

Risk

Risk-Reward Analysis


S

SD

SEM

Sample Size

Sample Space

Seasonality

Secondary KPI

Segmentation

Selection Bias

Sensitivity

Sequential Monitoring

Sequential Testing

Serial Testing

Severity

Sidak Correction

Sigma

Significance Level

Significance Test

Significance Threshold

Significant

Split Test

Standard Deviation

Standard Error of the Mean

Statistic

Statistical Design

Statistical Model

Statistical Power

Statistical Significance

Statistically Significant

Subgroup Analysis

Sufficient Estimator

Superiority Test

Survivorship Bias


T

T-test

Test Duration

Test Group

Test Of Significance

Theta

Transaction Rate

Treatment

Two-Sided Hypothesis

Two-Tailed Test

Two-sided Test

Type I Error

Type II Error

t Score


U

UMP

Unbiased

Unbiased Estimator

Underpowered Test

Uniformly Most Powerful Test


V

Variance

Variant


Z

Z-test

z Score


Glossary Index by Letter

If you have a suggestion for an A/B testing term we should add to this glossary do not hesitate to contact us.