## Statistical Significance Calculator - Video Transcript

In this video we present the Statistical Calculators in Analytics-Toolkit.com.

We’ll begin with our statistical significance calculator. Calculating statistical significance is a fundamental part of any internet marketing analysis or online testing activity. Without it you could easily fall prey to the noise in your data and mistake a false positive for a real result and vice versa.

Our calculator is really easy to use: you input the sample size, which can be the number of users, sessions, or ad impressions. Then you enter the conversion rate for the metric of interest, be it goal conversion rate, e-commerce transaction rate or CTR. It will then calculate both statistical significance and confidence intervals.

Our tool supports different levels of confidence thresholds and non-inferiority designs.

You can add and remove variants as you want. You can also do a bulk input where you can copy/paste data from a spreadsheet and it will automatically be populated into the calculator. We apply the Dunnett's correction to compensate for the multiple testing problem of many variants versus a control.

Non-binomial metrics are also supported, such as average revenue per user, average order revenue, and metrics like average pages per session and average session duration. The data entry process is a bit different, but just as straightforward.

If you are working with multiple KPIs and need an adjustment for multiple comparisons, then you can use our multiple comparisons calculator by entering either p-values or percent significance, and our calculator will apply the Šidák correction or the False Discovery Rate correction based on the method of Benjamini–Hochberg–Yekutieli.

Our third calculator is a sample size calculator. It lets you estimate how much traffic you’ll need to commit to a test in order to be able to reliably detect an improvement of a given magnitude.

Finally, we have an effect size and power calculator, which will display the power function of a test based on the number of users and a significance level required. Thus, you can check what power you have at different possible true lifts.

All the tools work under the assumption of a fixed sample size, determined in advance. For scenarios where sequential analyses are required, you can check our AGILE testing calculator.

The statistical calculators presented are essential for planning tests and analyzing data in marketing and A/B testing as practiced in conversion rate optimization, landing page optimization and e-mail marketing.

**Close [X]**