What is A/B Testing?

A/B testing, also referred to as “split testing,” is a marketing technique that involves tracking the performance of two different versions of something in order to determine what variables your audience responds to.

You can A/B test all sorts of things.

You could show 50% of your site visitors one version of a page while the other 50% see another version.

You could A/B test the language you use for your email opt-in form to track which is more effective.

You could test sending an email in the morning vs. in the late evening.

A/B testing is your chance to play mad scientist.

Today, we’re going to be focusing on running an A/B test on email subject lines in MailChimp.

How-To Guide: A/B Testing Email Subject Lines

One of the most important things to A/B test is the email subject lines on your marketing email campaigns.

The #1 thing that’s going to inspire your subscribers to view your email is the subject line, so its importance can’t be overlooked!

When running an A/B test on email subject line, you’ll write two different subject lines, and select how many subscribers (chosen at random) will receive one of the 2 versions. Some will receive subject line “A,” some will receive “B,” and the bulk of your list will receive the winner – as defined by superior open rate or click rate.

MailChimp takes care of the calculations and sends the winning version automatically.

Here’s how it’s done.

STEP 1: Create a new campaign

To get started, head to the “Campaigns” page on MailChimp and click “Create Campaign.”

STEP 2: Select “A/B Testing” as the campaign type

You may have been selecting “Regular campaign” in the past, which only includes 1 variation of the email.

To get an A/B test going (2+ variations), just select “A/B testing campaign” as your campaign type.

STEP 3: Select a list to email

Like with any other campaign, on this page select who you want to receive your email.

STEP 4: Design your A/B test

This next page is where you can set the parameters of your A/B test. Let’s look at the different elements to creating an effective test.

1. Test Variable – This is the area of your email you want to test. For this example, we want to select “Subject line.”
2. Distribution – Select what percentage of your total email list will receive variations of the test. We set this anywhere from 20-50%. If you select 50%, 25% will receive subject line “A,” 25% will receive subject line “B,” and the remaining 50% will receive the winner of the A/B test.
3. Winning Metric – Do you want the winner to be determined by open rate or click rate? We suggest going click rate, as that’s what we’re really after with most email marketing campaigns. Here is also where you set when the test will end, declaring a winner and emailing the rest of your list with the winning subject line. MailChimp says don’t set it below 4 hours for best results, but we won’t tell if you do.
4. Summary – Check out the hard numbers on this summary panel to monitor how many recipients will be sent each combination before you make a decision.

Here’s how one of our campaigns looks with all the details selected:

STEP 5: Write two subject lines to test

Finally, we’re ready to enter the actual content that will be tested. Use this opportunity to try out whatever you want with your email subject line.

Want to see if your audience responds to emojis in the subject line? Go for it. Want to try teasing the content of the email vs. stating it clearly? Let’s see what happens.

One of the great parts about A/B testing an email subject line is that you determine just how many subscribers will receive your test. This gives you the ability to test out ideas that may be too risky to email the full list with, lest they all hate it, don’t open it, or even unsubscribe.

Here’s our “A” and “B” subject lines for a recent email announcing a new feature to our wall preview tool:

STEP 6: Review your work before sending

On the final page of the MailChimp email workflow, you’re able to see a nice and succinct summary of your work. Review these details carefully to make sure you have set your test up correctly before scheduling or sending it.

STEP 7: Track results carefully

After your A/B test has been sent, you can keep an eye on it via the “View Report” button from the Campaigns page.

On the Report page you’ll find how the two emails are performing in terms of open and click rate, and you can watch as a winner is decided.

Should you not be happy with how the test is going, you can at any point during the test manually choose which version of the email you want to go to the remainder of your list.

Why cover A/B testing now?

The holidays are rapidly approaching and we really want to equip our readers with as many tools as possible to take advantage of this uber-profitable time in the art business.

Email marketing is a lot like a game of golf: You have to have a bag full of clubs to properly play the course, and A/B testing is a 7 iron.

So I am glad that we have covered how to do it MailChimp, yet I still owe you a rant. So here it is:

A/B testing for most people is a complete waste of time!!!!

The first time I heard the guy from Social Triggers say that I was taken aback by it as well. Then I thought his rant through, and I came to completely agree with it.

A/B testing is a complete waste of time for most people because they don’t have enough traffic.

I know, I know: “It’s scientific testing, it has a fancy name, it’s advanced marketing, it’s likely a panacea for all that woes my emerging business.”

Meh.

For most people out there, the most important thing for their business, especially if you are on our platform (we already did the a/b testing for you, the site WILL convert!), is not A/B testing, it’s getting traffic to your website.

100% of your efforts should be focused on driving more traffic to that website.

An Example

Let’s say you own a restaurant and it’s Friday night.

Currently, your restaurant has two diners in it. Ok. So as the owner what should you be doing right now?

Putting out one set of blue napkins and one set of red napkins and seeing which table the patrons sit down at?

Your damn restaurant is empty.

You ought to be out front with a bull horn, a large hook, and a whistle-making noise to drive folks into your restaurant…or whatever other creative strategies you have in place to drive traffic.

I think that metaphor describes it perfectly. Wasting time on A/B testing is a trap that is so easy to fall into.

Then why post about it here and advocate you do it in MailChimp?

Because A/B testing emails in MailChimp takes 30 seconds.

There are tests that use a lot of bandwidth, time, and resources – and these may be appropriate for the very large business with a very large sample size to draw conclusions from. For everyone else, be on the hunt for tests like the subject line test – very fast, easy to integrate into an existing process, and provides some quick, snappy results.

So, what size list is appropriate for this test? Let’s say if you have over 200 on your email list it’s worth giving it a shot.

If that’s you: go for it.

If that’s not you, then get some more patrons into the restaurant! But don’t worry, we’ll be back soon with some great techniques to do just that.