A/B Testing

With A/B Tests, also known as experiments or split tests, you grow your business and maximize your success by comparing different versions of your website or app to detect the most successful version that increases your sales, revenue, conversions, pageviews, and more.

A/B Testing is a premium feature which is included in our Cloud-hosted Business plan or you can purchase it on the Matomo Marketplace if you self host Matomo On-Premise. Learn more about all the benefits and features of A/B Testing.

Viewing Experiment Reports

Go to Matomo (Piwik). In the left menu click “Experiments” and then select an experiment of your choice. The reporting UI displays the currently running experiments as well as the finished experiments. If an experiment does not appear in this list it is either not running yet, or it has been archived.

Each experiment page consists of four type of different reports:

• A summary of your experiment details.
• An evolution graph letting you choose your variations and your success metrics to see how they evolve over time.
• An overview letting you easily compare all of your success metrics for each of your experiment variations.
• A detailed report for each of your experiment success metrics showing you which of the variation is currently performing best and how likely the change is actually statistically significant.

Managing your Experiments

To get to the “Manage Experiments” screen click on the user icon in the top right. There will be a new menu item “Experiments” in the left menu. They can be edited by anyone having at least ‘Write’ or ‘Admin’ access for a specific website. For each website or app, different Experiments can be created.

Creating an experiment

To create an experiment, click on “Create new experiment” in the bottom left. The form will ask you to enter some basic information to get you started with your experiment quickly.

During the whole process of creating and running your experiments, Matomo will show you help text explaining what each element means. This is especially helpful when you have not run experiments before.

Adjusting an experiment to your needs

Once you have created an experiment, you can adjust it to your needs. We recommend to first have a look at your Success metrics.

Comparing performance of different Pages URLs

In case you want to experiment and compare the performance of different pages URLs, you can create your experiment where you specify a Redirect Page URL for each variation.

When you create your experiment, under the section “Redirects”, for each variation (including the “Original” variation) you can can enter the Page URL to redirect to and test.

Here is how it works: when users land on the original page URL, they will enter the experiment (and be assigned one of the Redirect URL variation). Then they will either 1) stay on the current page (if the Original variation was picked), or 2) be automatically redirected to the Redirect URL they were assigned.

When you create an experiment using “Redirect URLs”, please make sure to include the Matomo Tracking code on all your pages including on the Redirect URLs pages.

For more information and ways to test different pages and use redirects in A/B Tests.

Starting your experiment

To start your experiment click on “Embed code” and embed the displayed code into your website, app, server or campaign. The experiment will automatically start running as soon as the first user enters the experiment. Alternatively, you can schedule a “Start date” by clicking on “Schedule”. To learn more about embedding your experiment into your project have a look at the A/B Testing developer guides.

Finishing your experiment

Once you have run your experiment for a while and came to a conclusion about your results, it is time to finish your experiment. An experiment can be finished by anyone who has at least Administration access for a specific website. To mark an experiment as finished, either click on the link “Finish experiment” on the top of your experiment report page (in the experiment summary), or click “Finish experiment” when editing the experiment in the “Manage experiments” section.

Once you have finished your experiment you should remove all code that is related to this experiment from your website or app.

You will still be able to view the report for your finished experiment as long as you have not archived it.

Archiving your experiment

When you archive your experiment, the reporting page and segments for this experiment will no longer be available in the Matomo reporting UI. This helps to keep the reporting UI clean and to focus on your new experiments. Once an experiment has been archived, you can still view it in the “Manage experiments” section by choosing the status “Archived” in the filter above the list of all active experiments.

An experiment can be archived by anyone having at least Administration access for a specific website either directly on the experiment report page in the experiment summary or when editing the experiment in the “Manage experiments” section.

Embedding / Tracking an experiment

To learn more about embedding an experiment into your project, have a look at our A/B Testing developer guides on our Developer Zone. You can run experiments on websites, servers, mobile and desktop applications and in campaigns.

FAQ

We compile a list of most frequently asked questions in the A/B testing FAQ.