What does "Independent Observation" mean?

What is an Independent Observation?

An independent observation is any data point in a set of data which is statistically independent from the rest. Independence means that its value is not influenced by the value of any other observation in the set. Independent observations are also not correlated, but the reverse is not true - lack of correlation does not necessarily mean independence.

For example, in an A/B test observations of user-level metrics are usually considered independent. The purchase revenue of one user does not depend on that of another. Even in this case there are certain violations of that assumption like a single person having more than one user account, or users who influence each other due to sharing the same physical space, or being colleagues, family members, etc.

On the contrary, observations of metrics based on sessions, pageviews, or ad impressions like ad CTR, page CTR, or conversion rate per session are usually not independent. The dependence they exhibit is due to the same physical person executing a series of actions. Each action then depends to an extent on whether something did or did not happen on the previous action. For example, a user who purchased during a prior session is much less likely to purchase in their current session. However, they might be much more likely to purchase after five or six more sessions.

The requirement for observations to be independent is key in defining the statistical null hypothesis of many commonly used statistical tests. Departures from independence can be established through mis-specification testing and if detected, a significance test that relies on this assumption is no longer adequate. The requirement for the observations being independent is often accompanied by the condition that they are also identically distributed.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.