A/B testing allows you to experiment with several variations of an insert's content to see which performs best. You can also add a control group - a group of users who don't receive the insert, to compare between the insert's performance of the various groups.
To start an A/B test, click on the A/B test toggle on the insert wizard setup step. You can name your variations or leave the default names.
On the Design and Content steps, toggle between the variations via the top right tabs. Configure each variation's design and content.
On the Audience step, select the audience split between the variations and the control group.
The insert dashboard will display the performance of the insert variations and the control group across the insert goals. Note the statistical significance indication next to the performance results - non-significance results are likely to occur by randomֿ. Only significance results are meaningful.
In case you chose to use a control group, the results of the variations will be measured against the results of the control group (we use it as the Baseline). If you see a Winner it means that this specific variation bit the control group.
If you are not using a control group, the results of all variations will be measured against the first variation (we use it as the Baseline). If you see a Winner it means that this specific variation bit the first variation.
Note that the Winner indication is per goal so theoretically there could be a variation which is a winner in one goal but doesn't win in another insert goal.
Tips for running an A/B Test
- It is usually recommended to change one aspect at the insert in each test to isolate the effect of this specific change (for example, the button's wording). However it is possible to test several changes at the same time.
- The more variations you use, the longer it takes to reach significant results
- Equal split of the audience to the variations will yield faster results (for example a 50:50 split will yield faster results than a 10:90 split).
- The bigger the audience included in the experiment, the faster the significant results.