Back to blog

12 Case Study Tips for A/B Split Test Email Campaigns in eCommerce


Mailigen AB test feature for email

Regular A/B split tests for email campaigns ensure more opens, clicks and conversions, according to Jorens Kovalovs from the Xnet online store. At the Mailigen breakfast, one of our most experienced clients shared with us (and you!) his A/B split testing experience. Learn from his case studies and follow his 12 tips to skyrocket your email performance!

How does A/B split testing improve email results?

Before we start discussing what kind of A/B split tests we can create in eCommerce, just a short reminder about what it actually is. An A/B test is a method used to check which of two subject lines, sender names, sending times, or email content elements is the most effective when sending it to recipients, thus obtaining the best result for your email campaign. For how to do that in the Mailigen system, read our article on how to send only the most effective email campaigns with A/B tests!

Hypothesis => solution => measurement => improvement => another hypothesis, etc.

This is the A/B split test scheme followed by Jorens. For example, you make a hypothesis that a green call-to-action button works better than a red one. Next, you send an A/B split test email campaign, using both buttons, thus clarifying the button with the most clicks. If the green button wins, it means this color is more often used in call to action buttons. Then you keep testing – on, for example, the words you put in these green buttons.

“The last step is particularly important: continuing to suggest new hypotheses in relation to the winner. Of course, one single A/B split test campaign will give a few percent increase in a campaign’s results. However, if you want a gradual improvement in results for all your email marketing performance, take a strategic and regular approach!” emphasizes Jorens.

A/B tests are performed in order to improve:

  • email opens
  • clicks
  • conversions

The last one is the most important, especially in eCommerce. If the email open rate and clicks increase, but conversions do not, you should dig deeper!

How do A/B split tests work in eCommerce?

Jorens is one of those people who likes digging deep. Very deep! That’s why his favorite webpage is Google Analytics. He has been engaged in A/B split testing since the moment he started email marketing, four years ago. Xnet is just one of numerous internet stores he does email marketing for.

He has tested all four elements offered by Mailigen A/B tests:

  • subject line
  • sender name
  • sending time
  • email content

The very diverse tests he has made on email content testing such elements as calls to action, email layout, images, colors, and even the background of the email.

The majority of tests are really simple ones that every marketer can do. Here are some of them you should try…

#1 Personalize email sender name

The sender name “Xnet internet shop” was compared to a personalized sender name “Martin from Xnet”. The second resulted in amazing opens and engagement, becoming the best Xnet campaign in 2014. Sounds too good (simple!) to be true? Next time, create a test for names and find out what works best for you!

#2 Choose a sender for each product group

Since its success with the personalized sender name, Xnet has moved on and started diversifying sender names. Now each product group has its own individual sender, so that the recipient gets to know right away what topic the email will be about. If you send several types of content, make use of several different sender names!

#3 Include recipient’s name in the subject line

Have you ever tried to include the recipient’s name in the subject line? Xnet A/B split test experience shows much better open rates when including a name merge tag in the subject line. However, Jorens suggests you do it only if you have information on all recipients’ names. If you don’t, make a segment and send this kind of subject line only to those with names.

#4 Add symbols to the subject field

Lots of Xnet A/B tests have been done including symbols in the subject field. Conclusion: yes, recipients notice them, react to change, and email opens even increase. But this is not the cause of conversion increase.

#5 Invest in email content

The only 100 % sure factor in dramatic conversion increase is content. Content is king! If the offer itself is good, convenient and really exciting, the campaign results will be good, independent of sender name, subject line, design or any other email element.

#6 Include several offers in one email

When working in eCommerce, it’s good to include more than one offer in your email campaign, says Jorens from Xnet. Besides that, don’t put the most popular products at the beginning of the newsletter. This will motivate recipients to look through the whole email down to the end. A/B split tests prove that most clicks appear on the top of the mail, so place there what you want them to sell. It’s the opposite situation in other industries, and the email campaign works the best with only one offer and one call to action.

#7 Highlight the numbers in eCommerce

Display of the price is email element no. 1 in eCommerce email campaigns. Several A/B tests done by Xnet have proved that a percent (for example, 40 % off) gives better results than a defined price (for example, prices starting from €20) because the discount as a percentage addresses a broad audience, not only those who are interested in the lowest price. Well, no matter how often discounts are used, they still work in marketing!

Mailigen AB split test for email

#8 Mix ordinary news with crazy campaigns

Not all campaigns must be super hits, admits Jorens. He suggests mixing big sales and crazy prices with calmer news and simple new offers without any discounts.

#9 Call to take a look, not buy

Most A/B split tests have, of course, been done regarding call to action buttons. Xnet experience shows that temperate Northerners prefer such buttons as “Take a look”, not “Buy”, because they are not sure about buying something right away. Well, this entirely depends on your audience, so make a test yourself! However, if such buttons as “Buy now” are used, you should definitely add a time limit to the offer in order to justify the urgency.

#10 Test colors

A/B tests have also been applied on buttons’ colors. As for Xnet, the green color turned out to be the most effective. As he admits, this is most probably because green is calmer and easier to understand. The colors have also been tested in other email elements, such as email background color. In that case, too, the green color turned out to be more successful. Which colors are the most engaging in your email campaigns?

Mailigen AB split test for email background

#11 Do the tests repeatedly

If you have proven your suggested hypothesis in the A/B test, do not consider it to be an absolute truth! This is particularly important in cases where the difference is not so big between the winner and the second version. Sometimes you need to explore which of the factors was determinant. Jorens from Xnet also suggests doing repeated tests for already tested parameters in order to really prove the hypothesis.

#12 Repeat, but not forever…

If you have discovered some elements in your A/B tests which work and provide good results, keep repeating this strategy from time to time. But not all the time! Remember, too, that “time to time” will not work forever. In time it won’t provide as good efficiency as in the beginning. So keep A/B testing and digging for new pearls!

Let’s get started

When used smartly, A/B split tests may result in significant email performance uplift! Hope you fully benefit from these tips and make the best of your email campaigns. To help you get started, we’ve prepared a video on how to create an A/B split test campaign step-by-step with Mailigen. If you need any other assistance, please contact our team!

12 Case Study Tips for A/B Split Test Email Campaigns in eCommerce
Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Pin on PinterestEmail this to someone

© 2010-2017 Mailigen. All rights reserved.