Experiments

Marketers need to know exactly how effective their actions are and how they affect their bottom line. When they try to evaluate their actions, they tend to attribute success or lack of success to various exogenous variables such as increased revenue around Black Friday.

Experiments bring a simple, yet very effective way to test your messages along multiple variables at the same time and find the highest-converting message.

In this article, you’ll learn about:

1. What Experiments are
2. Inspiration for what you may want to test with Experiments
3. How to run Experiments
4. How to run A/B Test With and Without an OptiMonk Campaign
5. How to track the monetary performance of your Experiment in Google Analytics
6. Things to keep in mind regarding Experiments
   
   

1. What are Experiments?

Simply put, you can now run controlled experiments to find out

• what message types work best,
• if your campaigns perform better than no campaigns at all,
• what segments have the highest conversion rates,
• what combination of messages bring the best results.

Experiments can be run including any existing campaign - both active and inactive on the same domain. You can pick any campaign to test against each other.

2. Inspiration for what you may want to test

1. A/B test message variants

If you want to test different design variants, CTAs, colors, copy or images in your messages, you can do that with Variant A/B testing within a campaign without running a full Experiment.

However, if you want to test different templates or message types against each other, you're at the right place. Experiments allows you to test

• gamification templates against standard list builders,
• lead magnet list builders against discount offers,
• teaser vs. no teaser campaigns
• and much more!

2. A/B testing segments & audiences

• test different geolocations agains each other eg. UK vs US
• test different Klaviyo segments & lists eg. new subscribers vs. potential purchasers
• test audience from different traffic sources eg. organic vs. paid traffic
• test segments with different cart values eg. low vs. high cart value

3. A/B testing the overall effectiveness of messages

Test messages against a control group to find out whether your campaigns have an impact on your revenue at all.

4. A/B testing complete sets/groups of campaigns

Test entire experiences to find the best combination of messages.

• Discount Code vs.(Discount Code + Discount Code Reminder)
• Discount Code vs. (Discount Code + Post-Purchase Feedback Popup)
• Discount Code vs. (Discount Code + Free Shipping)
• Free Shipping Bar vs. (Free Shipping Bar + Discount Code)
• Welcome Back Recommender vs. (Welcome Back Recommender + Discount Code

 

3. How to run Experiments

1. First, log in to your OptiMonk account at https://app.optimonk.com/login/en

2. Select the Experiments icon from the left navigation bar.

3. Here, you’ll see all your previously created experiments - if you have any - listed by their names and domains. If you haven’t run experiments before, don’t be surprised to find this page empty. To launch your (first) experiment, click on New Experiment in the top right corner.

4. First, select which domain you’d like to run the experiment on.
⚠️ Keep in mind that an experiments can only run a single domain a.k.a. you cannot test two campaigns running on two different domains within the same experiment.

5. Name your experiment to help you identify it later.
💡 Here’s a hint: include the domain and the subject of your A/B test in the name for easy identification. For example: US vs. UK segment | List builder | Sissora.com

6. Add campaigns to display to different visitor segments by clicking on Add campaign and browsing your existing campaigns.

⚠️  If you add an active campaign to an experiment, you’ll receive a small notification that this campaign will be inactivated and activated again once you start the experiment. The reason for this is that a campaign cannot run independently from an experiment.

⚠️  You can create up to 5 visitor groups.

7. Once you selected which campaigns you’d like to test against each other, determine the traffic share between the campaigns. You may want to display both campaigns in the same ratio to your visitors, but you may want to show Campaign A to most of them and only test Campaign B with a small traffic share. This option allows you to run tests with a lower risk.

8. To launch your experiment, hit on Start. Once you launch an experiment, if you wish to end it, you’ll have to do it manually.

9. You can end your experiment any time by clicking on End.

10. Once your experiment has been completed, you can view its results on the Experiments page.
⚠️ Keep in mind that the stats here only reflect the results achieved within the timeframe of the experiment. Further data can be measured in Google Analytics.

 

4. How to run Experiments With and Without an OptiMonk Campaign

In some cases, we would like to measure and see the additional revenue we can achieve with the help of the OptiMonk Campaigns. We can measure this with the help of Experiments, where

• one Group will contain the Campaign itself,
• while the other Group will represent those visitors that do not see our Campaign.

We will be able to track the results in Google Analytics:

Here you can see even more detail:

We will need a benchmark for the measurement, where we can see how much our visitors spend when they do not see any OptiMonk Campaigns. We will need two different groups for this measurement: one to which we will add a campaign that will show up to half of our visitors (since we are running an A/B test with 2 groups). The other Group will not contain a campaign, this will be the benchmark.

Thanks to the two Groups, we will be able to use Google Analytics to compare the spending of our visitors who had seen our OptiMonk Campaign and the visitors who did not see our OptiMonk Campaign. With these measurements, we will be able to tell without any doubt if our Campaign is profitable.

Here are the steps on how to run Experiments With and Without an OptiMonk Campaign:

1. First, log in to your OptiMonk account at https://app.optimonk.com/login/en

2. Select the Experiments icon from the left navigation bar.

3. To launch your experiment, click on New Experiment in the top right corner.

4. First, select which domain you’d like to run the experiment on.

5. Name your experiment to help you identify it later.

6. Add a campaign to the Control Group by clicking on Add campaign and browsing your existing campaigns.

⚠️  If you add an active campaign to an experiment, you’ll receive a small notification that this campaign will be inactivated and activated again once you start the experiment. The reason for this is that a campaign cannot run independently from an experiment.

7. Leave the other group empty by not adding any campaigns to it.

8. To launch your experiment, hit on Start. Once you launch an experiment, if you wish to end it, you’ll have to do it manually.

9. You can end your experiment at any time by clicking on End.

10. You can track the monetary performance of your Campaigns in Google Analytics by following the steps below.

5. How to track the monetary performance of your Experiment in Google Analytics?

You can track the performance of the groups of your Experiment in terms of profit with the help of Google Analytics. There are 2 ways to compare the results:

1. Create & compare segments: This allows you to dive into details

2. Create a dashboard: This allows you to monitor the results easier through a longer period

Create & compare segments

1. First, log in to your Google Analytics account, select Behavior on the left, and go to Events / Overview, then click on OptiMonk.

2. Select Event Label.

3. Pick which A/B test results you want to compare:

As you can see the labels are [experiment] + [experiment group name]. Choose the groups you want to compare. In this example, we will compare “Test With and Without an OptiMonk Campaign, group: Control Group” and “Test With and Without an OptiMonk Campaign, group: Group A”.

4. Click on Add Segment, then click on New Segment.

5. Create a segment by following the steps below.

1. Add a name, which should be the Event Label's name

2. Choose Conditions

3. Choose Event Label as a filter

4. Choose the label you want to segment by

5. Click on Save

Then repeat it for the other group.

6. Now you can see both reports with the 2 new Segments under Conversions, Ecommerce, Overview.

 

Create a dashboard

1. Go to Dashboards under Customization, then click on Create.

2. Choose Blank, then give it a name and click on Create Dashboard.

3. Add a widget like the one below, one widget for each group:

4. The result will look like this:

That's it! :) Now you can track the monetary performance of your Campaign Variants in Google Analytics.

6. Things to keep in mind regarding Experiments

1. If a campaign is part of an experiment, it will be indicated with a small icon on the campaign list page.
2. Once a campaign is added to an experiment, you cannot inactivate it from the campaign list.
3. Experiments can be run only once and cannot be restarted.
4. A given campaign can only be part of one experiment at once. Once that experiment has been completed, it can be added to a new experiment freely.
5. Users can only add campaigns to experiments that run on the same domain a.k.a. the domain of campaigns and the domain the experiment must match.
6. Results of each experiment are displayed on the experiment’s page. The statistics shown here refer only to the results achieved during the time window of the experiment. All-time performance of campaigns is displayed under Campaigns or Campaign Analytics.
7. Detailed data on experiments such as average order value, number of orders placed, revenue attributed to a campaign can be tracked in Google Analytics. OptiMonk shows data on impressions, conversions and conversion rates exclusively.

That's it! If you have any further questions or need any help, just let us know at support@optimonk.com and we would be happy to assist you :)