AGILE A/B Testing Calculator

An advanced statistical calculator for A/B & MVT tests. You can now run A/B tests up to 80% faster, thanks to the AGILE statistical method! Enjoy the flexibility to monitor results on accruing data while maintaining control over false positives and false negatives. Ideal for testing in conversion rate optimization, landing page optimization, e-mail marketing, SEO, PPC. To use this tool, you need to sign in to Analytics-Toolkit.com. Sign In or Free Trial.

Maximize the ROI of your A/B tests!

Differentiate between noise and data, errors and real results while running tests as quickly as possible! With this A/B testing calculator you get:

  • 20-80% faster A/B & MVT tests versus equivalent fixed sample designs
  • Flexibility to perform interim analyses and evaluations
  • Early stopping for both efficacy & futility (lack of positive effect)
  • Control of the tolerance for false positive and false negatives
  • Automatic retrieval of test data from Google Analytics®
  • No more illusory A/B testing results!

AGILE A/B Testing Calculator

Close [X]



AGILE A/B Testing Calculator - Video Transcript

Wouldn’t you like it if you could run A/B tests several times faster than current best-practice methods allow?

Don’t you want to be flexible in monitoring your tests as they go, without biasing your statistics?

Do you want to identify and stop futile tests quickly so you can focus on more promising ones?

You can now do all the above with the help of the AGILE A/B Testing Calculator.

It is based on the AGILE Statistical Method for conducting A/B tests in Conversion Rate Optimization, Landing Page Optimization, and other areas where scientific experiments can be applied to drive better business results.

The AGILE A/B Testing Calculator allows you to run A/B & multivariate tests 20-80% faster than traditional fixed sample designs and gives you the flexibility to monitor results as they accrue. It is based on proven and time-tested statistical procedures used in medical research.

It can help identify better versions quicker so you can implement them sooner, improving your bottom-line. It also helps detect unpromising tests quickly allowing you to cut revenue loses and the time committed to them.

All the while the tool helps you maintain rigorous statistical control over false positives and false negatives.

The AGILE calculator is designed to be used alongside an existing tool for delivering A/B or MVT tests. The test data is then entered manually or syndicated automatically, for example from your Google Analytics account.

To experience AGILE sign up for a non-obligatory free trial and explore our free white papers and articles on AGILE A/B testing. Start running more efficient A/B tests today!

Close [X]

5 reasons to use the AGILE A/B testing calculator now



#1. Faster A/B and multivariate tests

Get an edge on your competitors and amaze your clients with significantly faster tests. Depending on what the actual difference between your tested variant(s) and control is, the gains can range from the moderate 20%, to a whopping 80%!

You can run tests and make decisions 20-80% faster than traditional best-practice methods, without sacrificing the rigourorous error control.

Comparing the performance of sequential AGILE A/B tests with classical fixed-sample statistical signifiance tests




An AGILE A/B test, stopped early for efficacy (winner declared)

An AGILE A/B test, stopped early for efficacy (winner declared)

#2. Flexible monitoring & decision making

Do you have to compromise between running a proper test and pleasing a client or HiPPo who demands to see and act on the results? Like, right now! Stopping a classical statistical significance test based on interim data would result in breaking the statistical assumptions of the test and is thus a big no-no.

No more compromise! Run flexible sequential A/B and A/B/n tests, periodically evaluating the data and making decisions to stop or continue testing. The stopping boundaries for efficacy and futility are based on your desired statistical significance and statistical power so you always have guidance on when to call a test a winner or a loser.





#3. Early stopping for efficacy & futility

Stopping rules (also Decision rules) are what allow sequential monitoring of test results and they guide your decision to stop the test. When running A/B tests, it is often difficult to tell when to stop the test, especially of the results seem unpromissing, but that is not an issue with our statistical calculator.

The efficacy stopping boundary will tell you when to stop a test and declare a statistically significant winner, while the futility stopping boundary will tell you when there is little chance for the test to turn out a winner. Thus AGILE allows you to minimize budgets wasted chasing unpromissing variants and quickly find real winners in order to capitalize on them, improving the ROI of your A/B testing efforts as a whole!

Decision boundaries of a sequential A/B test




Correct and incorrect conclusions (errors) when interpreting A/B test data

Correct and incorrect conclusions (errors) when interpreting A/B test data

#4. False positive & false negative control

We emply best-practice statistical procedures from medical testing to allow flexible monitoring without sacrificing error-control.

No more compromise! Run flexible sequential tests, periodically evaluating the data and making decisions to stop or continue an A/B test. The stopping boundaries for efficacy and futility in the calculator are based on your desired statistical significance and statistical power so you always have guidance on when to call a test a winner or a loser.





#5. Automatic A/B test data retrieval

There are two ways to get data into this A/B testing statistical calculator. One is manually entering of summary data. It is a tedious process, even with our bulk input method, but it can be used with any A/B testing system.

Secondly, if your A/B testing delivery software syncs data to Google Analytics® (many industry leaders do), our calculator can automatically extract it using the Google Analytics® API. This happens in a completely automated fashion and you will get e-mail notifications when a test is ready to be stopped.

Automatic test data retrieval

Close [X]



AGILE A/B Testing Calculator - Video Transcript

Wouldn’t you like it if you could run A/B tests several times faster than current best-practice methods allow?

Don’t you want to be flexible in monitoring your tests as they go, without biasing your statistics?

Do you want to identify and stop futile tests quickly so you can focus on more promising ones?

You can now do all the above with the help of the AGILE A/B Testing Calculator.

It is based on the AGILE Statistical Method for conducting A/B tests in Conversion Rate Optimization, Landing Page Optimization, and other areas where scientific experiments can be applied to drive better business results.

The AGILE A/B Testing Calculator allows you to run A/B & multivariate tests 20-80% faster than traditional fixed sample designs and gives you the flexibility to monitor results as they accrue. It is based on proven and time-tested statistical procedures used in medical research.

It can help identify better versions quicker so you can implement them sooner, improving your bottom-line. It also helps detect unpromising tests quickly allowing you to cut revenue loses and the time committed to them.

All the while the tool helps you maintain rigorous statistical control over false positives and false negatives.

The AGILE calculator is designed to be used alongside an existing tool for delivering A/B or MVT tests. The test data is then entered manually or syndicated automatically, for example from your Google Analytics account.

To experience AGILE sign up for a non-obligatory free trial and explore our free white papers and articles on AGILE A/B testing. Start running more efficient A/B tests today!

Close [X]



Need more information?

How to use this statistical calculator?

The tool is designed for usage in conversion rate optimization (CRO), landing page optimization (LPO), e-mail marketing, SEO, and PPC experts in mind. The basic flow is such: design a test, run the test and input data as it accrues, and finally analyse results. We support automatic data retrieval of A/B test data from Google Analytics, given that you use custom dimensions to differentiate between users or sessions exposed to the different test variants.

The tool is quite intuitive to use and has plenty of contextual help on each screen. This post on designing an AGILE A/B test is a good starting point for understanding the different parameters and trade-offs in using the calculator's options.

Educational Resources

Check out our Free White Paper, with a detaled description of the AGILE statistical approach to A/B testing, complete with simulations.

Posts on AGILE A/B Testing on our blog. Practical advice on applying the AGILE method, as well as details on the major fundamentals. Make sure to also check our posts on A/B testing and Conversion Optimization in general.