What does "Argument from Coincidence" mean?

What is an Argument from Coincidence?

The argument from coincidence is a particular logical interpretation of the outcome of hypothesis testing in frequentist inference. It is basically a statement that if we apply a procedure which has a very low error probability we can interpret particular outcomes of this procedure as non-erroneous to just that extent based on that fact.

A form of the argument of coincidence can be traced back at least to Ian Hacking's "Representing and Intervening: Introductory Topics in the Philosophy of Natural Science" wherein it is argued that if two independent methods produce sufficiently similar measurements this represents a strong argument for the accuracy of those measurements as it would be a "preposterous coincidence" if they did so given the true value is different.

In the setting of an online controlled experiment an argument from coincidence allows us to argue from error: there is evidence an error is absent to the extent that a procedure with a very high capability of signaling the error, if and only if it is present, nevertheless detects no error. In an A/B test if we consider a risk of a type I error of 5% acceptable, after observing the result of a significance test that produced a p-value of 0.01 we can argue that we are not in error in concluding that the treatment we tested is more effective than the control due to the testing procedure having probability higher than 95% to register that, had if been true, yet it did not. Naturally, the actually observed 99% probability should be reported as being more informative.

References:
[1] Mayo D. (2018) "Statistical Inference as Severe Testing"

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.