If you’ve been following me and reading my posts, you know that I’m not a big fan of subject line testing.
It’s not that testing subject lines isn’t a good thing to do. It’s because many email marketers ONLY test subject lines, because it’s pretty much the easiest thing to test.
Subject line tests can boost performance, but they rarely provide a significant boost with a single test. Which is why I use them as part of a comprehensive testing plan, but I use them sparingly.
This case study is the exception. It’s from late 2020; this subject line test was one of nearly 20 tests, of all different types, that we did over the course of this 4-month campaign.
The client is a B2B company that provides corporate and employee holiday gifts, often personalized with their customers’ organization names and/or logos. The audience here isn’t buying a single gift for a spouse or a couple gifts for their children. They’re buying anywhere from 10 to 1,000 or more gifts for their customers and/or employees.
We used the top performing email messages, using revenue-per-email (RPE) as our KPI, from the 2019 holiday season as the starting point for our 2020 holiday campaign. As a result, the 2019 version of this email was the control; after analyzing the message and coming up with hypotheses to improve performance, we decided on a test.
There are a lot of possible things to test here, and we went on to do more testing on this email in subsequent sends.
But for this one, we decided on a subject line test.
Why?
Because I saw a real opportunity for a lift in the control subject line. Also because we were doing a resource-intensive test for another send the same week that was to be deployed, so it was a good time to do a simple test.
The control subject line here was “Holiday Gifts Under $10.”
If you’re going to do a subject line test, you MUST start with a hypothesis.
A hypothesis is a supposition of proposed explanation made on the basis of limited evidence as a starting point for further investigation. Basically it’s a theory of how you can improve performance, based on some sort of data.
This is important, because if you don’t begin with a sound hypothesis of why what you’re testing will improve performance, you’re less likely to see the boost you’re looking for.
My hypothesis: Leading with the most important part of the subject line, ‘Under $10,” will boost revenue-per-email (RPE) by at least 10%.
This hypothesis is based on my past experience, with a slight twist.
My standard rule on subject lines, based on my past experience, is to make sure the key message appears in the first 25 characters. I’ve got a solid track record of success following this rule (here’s just one example, where we got a 14% lift in RPE).
But here’s the thing. The control subject line is only 23 characters – so the key message is already in the first 25 characters.
In this case, I decided to test whether putting the key message right at the start of the subject line would make a difference. Would it boost performance over and above the performance boost we see with the 25-character rule?
And you know from the title of this post that it did.
Here’s the test we did:
Simple right? Cost was pretty much nothing. But the results were huge.
You can see the results – here we are using revenue-per-thousand-emails-sent (RPME), to make the numbers a little easier to manage. If we used revenue-per-email (RPE) the variance would be the same, but the numbers would be small ($0.036 and $0.058).
Our key performance indicator here is RPME, and the test generated an RPME that was 63% higher than the control. Based on this, it’s clear that the test is the winner. We’ve proven the hypothesis – putting the key message at the start of the subject line, even if the entire subject line is less than 25 characters, boosted performance.
But there’s more we can learn from the metrics.
If you look at the conversion rate, you’ll see that the test lagged the control here, by 9%. The control drove more unique transactions.
But the average order value (AOV) for the test was much higher than it was for the control, $407 (rounded up) compared to just $228. That’s a lift of 79% that the test subject line provided.
The variance in AOV is interesting – it’s even larger than the RPME variance (79% vs. 63%).
Now let’s look at the ‘usual’ email marketing metrics.
A lot of email marketers use open rate as their KPI when they are testing subject lines – and they declare a winner based on that. As you can see here, that would have been a big mistake, since the control open rate (5.6%) was higher than the test open rate (5.3%), even though the test brought in 63% more revenue.
We see a similar story with the click-through rate (CTR). The test lagged the control by 16% here but ended up bringing in 63% more revenue. As a result, we see that, like open rate, CTR isn’t a reliable KPI.
This data supports something I state frequently – that your KPI should always be a business metric (like RPE or RPME), not a diagnostic metric (like open rate or CTR). This is true no matter what element of the email you are testing.
What happened here? Few people opened the test, fewer people clicked on the test, but the test drove 63% more revenue than the control.
I think that the test subject line resonated with people who wanted to reward their employees and customers for the difficulties that the pandemic caused in 2020 but were very price sensitive (since many companies took a hit financially from COVID-19). There were fewer of them that opened the email message, but they were more motivated to purchase.
Having “Under $10” right at the start of the subject line grabbed attention in the inbox and set this email apart from other holiday emails. So, the people who opened the email and read it were highly motivated to purchase.
Try a hypothesis-based subject line test with your next email campaign – and let me know how it goes!
Be safe, stay well, peace,
jj
Photo by Pablo Arroyo on Unsplash
