Manual A/B test data entry

Learn how to use the manual data entry mechanism for both single evaluation and sequential evaluation tests planned using the A/B testing hub.

One of the ways to enter data for an A/B test planned and analyzed using the A/B testing hub is to manually input it through the interface. While this is perfectly fine for single evaluation tests, it may be cumbersome for sequential evaluation tests with multiple data observations. In the latter case, other data entry mechanisms should be preferred, for example using our Custom API or a Google Analytics link.

Entering binomial data

If the test uses some kind of event rate, e.g. a conversion rate or purchase rate, the data is that of a binomial metric. Entering it requires one to know the sample size (number of users or sessions, depending on what was chosen in the test creation wizard) and either the number of events or the rate itself. When entering a number of events or a rate, the value for the other field will be calculated automatically on the fly, so you do not need to fill both fields.

data entry for a binomial experiment

In the screenshot above, the user is entering data for an experiment on a binomial metric with sequential evaluation (a.k.a. AGILE A/B test). The number of the current analysis is displayed for convenience. Click on the "Add data" button to submit the information to Analytics Toolkit.

data entry success screen

If the data was recorded successfully, the above message will appear in case of a sequential evaluation test. Unless you are re-analyzing an old test for some reason, it is strongly advised to proceed by using the "Go to data analysis" button. This will load the test results screen, at which point the data is evaluated for boundary crossing and an appropriate action is suggested. If the efficacy boundary was crossed, the test will be stopped and no further data can be entered. If the futility boundary has been crossed, the test will be stopped if it has a binding futility boundary, or an option will be given to the user to stop it if it has a non-binding futility boundary.

If the test is a single evaluation design, a single button prompting you to see the data analysis will appear on successful data entry. No further data can be entered for a single evaluation test as it has been planned for a single observation only.

Entering non-binomial data

Non-binomial or continuous data needs to be entered if a test has been designed in which the primary metric is a non-binomial metric. Prominent examples of such a metric is average revenue per user (ARPU). Other examples include average sessions per user, average revenue per sessions, average order value (AOV), average session duration, and so on.

Direct data entry

Direct entry of non-binomial data is recommended only if you have software which has already automatically computed the number of users/sessions, the arithmetic average, and the standard deviation in each test group. Otherwise it is recommended to use the file upload function with raw data from which Analytics Toolkit will compute all three metrics.

direct data entry for a non-binomial experiment

To proceed, simply enter a value in each of the fields (all are required). Then the "Add data" button will become active and you can submit it. For next steps see the "Successful data entry" section below.

Data entry via file upload

The recommended method for manual data entry of non-binomial data is via a file upload. The form allows you to select a file to upload or to drag and drop a file from your file explorer interface.

Supported file formats include Excel spreadsheets (.xlsx), comma and tab delimited files (.csv, .tsv). Each individual value should be on a separate row. The control data should always be in the first column. Subsequent columns should contain data about the variants: column two should contain data for variant one, column three data for variant two, etc. depending on the number of variants in the test.

You can download these example files for reference on the correct formatting of the uploaded files: example XLSX, example CSV.

non-binomial data file upload form

Once selected, the file is automatically parsed and all fields in the form will be filled automatically in case of successful parsing, as shown below.

file uploaded and parsed successfully

If there was an issue during file upload or parsing, an error message will be displayed with pointers on what the error was and how to address it. If the issue is with file parsing, please refer to the example files referenced above and check whether the formatting of your files matches the examples.

Successful data entry

data entry success screen

If the data was recorded successfully, the above message will appear in case of a sequential evaluation test. Unless you are re-analyzing an old test for some reason, it is strongly advised to proceed by using the "Go to data analysis" button. This will load the test results screen, at which point the data is evaluated for boundary crossing and an appropriate action is suggested. If the efficacy boundary was crossed, the test will be stopped and no further data can be entered. If the futility boundary has been crossed, the test will be stopped if it has a binding futility boundary, or an option will be given to the user to stop it if it has a non-binding futility boundary.

If the test is a single evaluation design, a single button prompting you to see the data analysis will appear on successful data entry. No further data can be entered for a single evaluation test as it has been planned for a single observation only.