A/B testing statistics done right

Significantly faster A/B tests

Get an edge on your competitors and amaze your clients with the significantly faster AGILE tests.

Benefit from winners earlier and cut losers as quickly as possible and increase the business returns from your A/B testing program. Depending on the actual difference between the tested variant(s) and control is, the gains from using the sequential evaluation methodology of the A/B testing hub (AGILE) can range from the moderate 20%, to a whopping 80% faster!. Real-life data show an average benefit of 26% for clients of Analytics Toolkit.

Get the benefits of sequential testing without sacrificing the statistical error control which would otherwise be compromised by what is known as peeking. The embedded calculation of statistical significance, confidence intervals, and other statistics are as rigorous as they come.

Evaluate data on the go

Run flexible sequential A/B and A/B/n tests, periodically evaluating the data and making decisions to stop or continue testing.

Have you found yourself having to compromise between running a proper A/B test and pleasing a client or HiPPo who demands to act on the results right this minute? Stopping a classical statistical significance test based on interim data would break the statistical assumptions and invalidate its error guarantees.

No more compromises! Get the flexibility and efficiency your A/B tests deserve with AGILE sequential testing. Periodically evaluate A/B and A/B/n tests with stopping boundaries for efficacy and futility which guide you when to call a test a winner or to cut a non-promising test short.

Robust statistics you can trust

Unrivaled statistical rigor means you can trust your A/B test results. Sequential evaluation tests using AGILE are based on the same methodology which was employed in the COVID vaccine trials - easily the most scrutinized experiments in human history. P-values, confidence intervals, and lift estimates are adjusted uniquely so they account for the sequential nature of the test. The statistical estimates have been checked through millions of simulations.

The A/B testing hub offers an integrated experience of planning and analyzing both single evaluation and sequential evaluation tests. This helps avoid many of the common pitfalls encountered in applying statistics in A/B testing. The instrument chooses the correct statistical methods and adjustments for you and applies them automatically giving you the much-needed confidence in the results.

Plan tests with optimal business impact

Plan ROI-optimal tests and remove the guesswork in choosing how long to run a test and what confidence to require.

Coming up with the right test plan can be a daunting task while the costs associated with getting a test plan wrong can be significant. Test for too long and you run the risk of exposing too many users to a variant which loses money, or to unnecessarily delay the implementation of a winner which means lost business revenue. Test for too short and risk a high probability of a false negative and therefore failure to implement actual winners. Similar issues accompany the choice of the confidence required to call a test’s outcome.

The A/B testing hub helps you plan tests that deliver optimal business returns without going through the typical difficulties. Given relevant business details related to the test at hand the system searches through the space of possible combinations of test durations and confidence levels until it arrives at a nearly optimal one.

Automate data extraction and reporting

There are three ways to get data into this A/B testing statistical calculator. First, you can manually enter summary data. A tedious process for sure, even with file uploads, but it comes with the benefit that it can be used with any testing system.

The second data input method supported by our A/B testing calculator is to use our simple Data API. It can work with any system for storing A/B test data, including custom-built ones, as long as you have the capability to code the API endpoint. It is nicely complitmented by our powerful Reporting API.

Third, if your AB testing delivery software syncs data to MixPanel® our calculator can automatically extract it using the MixPanel® API. This happens in a completely automated fashion, and you will get email notifications when a test is ready to be stopped.