Conventional wisdom is that personalizing an email message will boost your bottom-line performance by boosting relevance. So it might seem like a waste of time to do a test and confirm that you see a lift when you personalize an email.
But it’s always a good idea to test. If the test version bests the control version, you can get a quantitative read on the lift. And if the test doesn’t best the control… then you’ll be thankful that you tested it before rolling out the change and depressing your performance for the foreseeable future.
Background
This client sells personalized products to organizations (they are B2B) – think the pens, flash drives, t-shirts, and other ‘freebies’ you see at booths at trade shows, or the holiday gifts you may have received from your employer or from a vendor. In both cases, the products you’re given are embroidered or printed with the name of the organization giving them away.
This was the fifth personalization test we ran with this client over a 3-month period. I’ll talk a little more about the first four tests at the end of this post.
Personalization Test #5
The product in this particular email test was a pen.
The control had an image of a pen with “Your Imprint Here” printed where the recipient’s organization name would be printed.
For the test we would use real-time personalization, at the time the recipient opened the email, to put their organization name on the image of the pen. This was the personalization we were testing.
Wireframes of both versions appear below. The gray area was the same for both versions (I’ve truncated the wireframes to save space); the only difference was in the product image, which is green here.
We broke the list into three segments.
We were using technology that pulled the company name from the domain in the email address. As a result, we needed to suppress from the test any non-business domains (think email addresses that end with yahoo.com or gmail.com). This was the first group.
After we isolated the non-business domains, we randomly split the balance of the list into two groups; one would be the control group and receive the control version of the email. The other was the test group, and would receive the test version of the email
Here’s the split:
The control and test groups each had more than 275,000 names in there, well above my minimum cell size of 20,000, which almost always allows me to get statistically significant results.
As always, we kept everything except the personalization the same, so we could get clear results.
Which version do you think won? Was it the test with personalization? Or the control without personalization?
I’ll give you a minute to think about it… then scroll down to get the answer.
Results
Here are the results!
That’s right. The control, which was not personalized, generated a higher average revenue-per-thousand-emails-sent than the test.
How many of you guessed correctly?
Our key performance indicator (KPI) here is revenue-per-thousand-emails-sent (RPME), because we are looking to optimize revenue.
Note: We used RPME instead of revenue-per-email (RPE) because for this client the RPE numbers were often very small. By increasing the magnitude it makes it easier to see variances – and the relative variance between the cells remains the same.
Here are the full results of the test:
As you can see, the RPME for the test was 43% lower than the RPME for the control version. If we look a little further we can see why the control bested the test.
It has everything to do with the average order value (AOV). The test version delivered an AOV that was 43% lower than the control version. As you can see in the column at the far right, the conversion rate for the two versions was exactly the same.
Now let’s take a closer look at the diagnostic metrics…
The test bested the control in both the open rate (this is a pre-MPP send) and click-through rate (CTR). It had a slightly higher open rate (0.7%) and a higher CTR (nearly 19%). But it still generated a lower RPME than the control.
The click-to-open Rate gives us a clean read on the body content of the email, as it adjusts the click rate for the variance in the open rate. Here you can see that the test bested the control by more than 20%. So of those who opened the email, 20% more clicked on the test/personalized version than clicked on the control/not-personalized version.
This illustrates something I’ve written about before: that CTR is not a good KPI if you’re looking to generate revenue. Because a higher CTR doesn’t mean a higher RPME.
Were we surprised by the result?
Yes. Surprised and profoundly disappointed. Because we truly believed that personalization would boost response – but this test proved us wrong.
The Previous Personalization Tests
As I mentioned, this was personalization test number 5. Why did we test personalization 5 times? Because the results were mixed and we needed to understand whether personalization was boosting or depressing response.
Here’s our personalization test journey:
- Test 1: 96% increase in revenue from personalization
- Test 2: 67% decrease in revenue from personalization
- Test 3: Tie; no sales from either the not-personalized control or the personalization test version
- Test 4: 22% decrease in revenue from personalization
So we tested again, described here, (#5) … and got a 43% decrease in revenue from personalization.
Take-aways
So, does this mean that you should not personalize your email messages?
No.
Does it mean you should test personalization before you go all in on using it?
Yes.
I’ll be writing a debrief on the full journey, and let you know where this client ended up with personalization, soon. Watch this space…
In the meantime, give this type of personalization a try with your program and let me know how it goes! Or let’s talk about creating a performance optimization testing program for your organization!
Be safe, stay well,
Photo by Shubham Dhage on Unsplash
