Statistical Calculators

Setting up and analysing A/B tests? Doing online marketing analysis? Our statistical significance calculator, sample size calculator, and power & effect size calculator help you distinguish insights from noise in your data. Use it now: Sign In or Start Free Trial.

Advanced Statistical Significance Calculator

If you are making data-driven decisions you need statistical calculators in order to mimize the possibility of wasted budgets or, to the contrary - missed opportunities. Without a significance calculator you can't differentiate between noise and data, chance and real results! With this tool you can make sure that the experiments you set up are good and that your decisions are based on statistically sound data.

Our statistical significance calculator is very advanced and excels where many free calculators will fail you. It properly handles testing multiple variants against a control by adjusting for the increased False Discovery Rate. The sample size calculator supports control over statistical power. All tools also supports non-inferiority tests.

  • Advanced Statistical Significance Calculators
  • Sample Size Calculator
  • Power & Effect Size Calculator
  • Bulk data input for ease of use
  • Supports revenue-based metrics, pages/session, session duration, etc.
  • Non-inferiority tests (sample size & significance)

Statistical Significance Calculator

Close [X]



Statistical Significance Calculator - Video Transcript

In this video we present the Statistical Calculators in Analytics-Toolkit.com.

We’ll begin with our statistical significance calculator. Calculating statistical significance is a fundamental part of any internet marketing analysis or online testing activity. Without it you could easily fall prey to the noise in your data and mistake a false positive for a real result and vice versa.

Our calculator is really easy to use: you input the sample size, which can be the number of users, sessions, or ad impressions. Then you enter the conversion rate for the metric of interest, be it goal conversion rate, e-commerce transaction rate or CTR. It will then calculate both statistical significance and confidence intervals.

Our tool supports different levels of confidence thresholds and non-inferiority designs.

You can add and remove variants as you want. You can also do a bulk input where you can copy/paste data from a spreadsheet and it will automatically be populated into the calculator. We apply the Dunnett's correction to compensate for the multiple testing problem of many variants versus a control.

Non-binomial metrics are also supported, such as average revenue per user, average order revenue, and metrics like average pages per session and average session duration. The data entry process is a bit different, but just as straightforward.

If you are working with multiple KPIs and need an adjustment for multiple comparisons, then you can use our multiple comparisons calculator by entering either p-values or percent significance, and our calculator will apply the Šidák correction or the False Discovery Rate correction based on the method of Benjamini–Hochberg–Yekutieli.

Our third calculator is a sample size calculator. It lets you estimate how much traffic you’ll need to commit to a test in order to be able to reliably detect an improvement of a given magnitude.

Finally, we have an effect size and power calculator, which will display the power function of a test based on the number of users and a significance level required. Thus, you can check what power you have at different possible true lifts.

All the tools work under the assumption of a fixed sample size, determined in advance. For scenarios where sequential analyses are required, you can check our AGILE testing calculator.

The statistical calculators presented are essential for planning tests and analyzing data in marketing and A/B testing as practiced in conversion rate optimization, landing page optimization and e-mail marketing.

Close [X]

Tool Characteristics

Automation N/A

Time-Saving N/A

Unique Tool Yes / No

API Only N/A

Expertise Level Novice / Experienced / Expert

How to use this tool?

There are actually several tools under the hood here. The first one allows you to compute how many users you would need to have a statistically significant result and a desired test sensitivity (statistical power). The second one let's you calculate statistical significance and confidence intervals that give you an estimate of the uncertainty in your data - that is, whether the observed differences between your variant and control can be explained just by the natural variance of the sample. Others offer even more advanced functionalities.

The statistical significance calculator supports multiple testing corrections and corrections for multivariate tests. Calculating statistical significance for revenue metrics is also a breeze (average revenue per user, average revenue per order, sessions per user, pages per session, average session duration, etc.). It is ideal for A/B testing in conversion rate optimization, landing page optimization, e-mail marketing optimization, and others.



Our clients about us

The value added to our agency by these tools is tremendous. I would say it provides Inflow with the best bang for the buck out of any tool we pay for. [ full testimonial ]

Chris Hickey
Director of Inbound Services, InFlow

From time-saving tools to quick and amazing support (in the rare case) it is a must have. Analytics-Toolkit is truly a must-have for any large analytics team. [ full testimonial ]

Thomas Bosilevac
President and Founder, MashMetrics