Micro Course part 3 – Were you right?

In this part, let me give you the results from our last lessons FUN split testing quiz.

The important take away here is that tests can only tell part of the story. THey also need to be statistically significant. You’re on highly dangerous ground if you make long term business decisions based on 1% or 2%, when the sample size was say 100 or 500. That means what 10 people did, or didn’t do, that day, changes what you do? Risky!

Another important takeaway here is that OPEN RATES did not necessarily mean “SALES”! Read the cliff notes 😉


SPLIT TEST 1: Travel Industry:
A: Time is almost up. Click below by 30th for your chance to own an iPad
B: Are we still welcome in your inbox? iPad prize for grabs until 30th November.
C: Are we still welcome in your inbox? Please reply by 30th November

The Split Test 1 Result:
Subject line C outperformed A with 143% increase in positive response. Subject line B was a close second

Split Test 1 Cliff Notes:
This campaign was the last email of a 3 email sequence sent to inactive travel customers who had not opened an email in over 9 months.

With inactive audiences a strong sell in the subject line is often a turn off. Questions tend to work better. So consider testing a question based approach for different segments of your audience.

SPLIT TEST 2: Webinars & video platform:
A: Live tomorrow: Next Generation Desktop Summit
B: Live today: Next Generation Desktop Summit

The Split Test 2 Result:
When sent to an in-house list, subject line A gave an increase in opens of 111.7% and 153% in clicks. However, it was B that gave more event registrations.

Split Test 2 Cliff Notes:
The subject line that generated the most interest didn’t generate the most conversions. In further details of the test, they also ran the same test to an “acquired” list. For this less engaged list the OPPOSITE performance was seen, with split test A generating the most conversions.

Proving that different audiences can and do respond differently. Just because someone else had a winning subject line it may not be a winner for you. Also with event/results metrics, you need a larger sample size to make it accurate.

SPLIT TEST 3: Insurance Agency:
A: $20 Million Dollar Homes Burned To The Ground
B: Protect Your Home from Wild Fires – Brush Fire Season is Coming

The Split Test 3 Result:
Subject line A won increasing leads by 65%

Split Test 3 Cliff Notes:

Subject line A invokes emotion and grabs attention rather than the simple sales line in subject line B.

The winning subject line reads like a newspaper headline, has elements of fear and intrigue that makes the reader go further. But while “attention grabbing” subject lines produce higher open rates, they can fail to produce higher click rates. Perhaps because the readers are less qualified to the offer? Another good reason to ensure you test and to determine winners by clicks or conversions rather than open rates!

So it’s advisable to try testing totally different subject styles and not just minor tweaks; Try testing; urgency, clarity, intrigue, emotion and using questions, in subject lines.

SPLIT TEST 4: Internet Marketer:
A: 1 simple secret to skyrocket the quality of everything you do
B: How I ‘hack’ excellence with 1 simple trick

The Split Test 4 Result:
Subject line B had a 12% higher click rate – but see the cliff note….

Split Test 3 Cliff Notes:
After checking the results, the sample size (2700) and uplift are not sufficient to be certain of a winner. There is weak statistical significance of 80%. The click difference could be just random variation and it’s misleading to draw conclusions and build future campaigns on any perceived insight.

There are many split test stories around the internet that lack the rigour needed for solid and continuous optimisation.


In our next lesson together we’re going to look at something INCREDIBLY IMPORTANT. How to create email traffic magnets to grow your list with the RIGHT people…

Watch out for it over the next couple of days.

All the best,

Miso from Email Exploder

About the author:

Tim Watson is an email marketing consultant who has over 12 years of specialist Email Marketing experience working with blue chip brands mainly in the US and UK.

In addition to strategic email consulting services, Tim is an email marketing trainer and frequently speaks at industry events. He’s an active blogger not just reporting the trends but providing thought leadership on the opportunities and challenges in the industry. As well as posts here he’s a regular contributor to the DMA Email Council and Smart Insights blogs, as well as a guest blogger for other marketing focused sites.

He is an elected member of the UK DMA Email Council, supporting the email marketing industry and Chairs the Legal and Best Practice hub of the Email Council, authoring and reviewing DMA whitepapers and best practice documentation.

Source: http://www.zettasphere.com/results-of-9-subject-line-ab-split-tests/