“Adding SMS to our abandoned cart program has more than tripled our revenue!”
Looking to add SMS to your marketing mix? SMS plays well with other channels – especially email. But if you really want to get a read on how well it’s performing, implementation matters.
I started working with this client about 9 months after they added a single SMS text message to their existing 3-effort email abandoned cart campaign, so I had 8 months of data to analyze to gauge the effectiveness. The initial results looked impressive.
A year-over-year (YOY) lift of 242% in revenue – that’s more than triple the revenue in 2022 that they generated in 2021.
When I looked deeper, it was clear that the new SMS effort was the cause of the boost. Revenue from the email messages was roughly steady, down just 1% from the prior year. The SMS message was responsible for 100% of the YOY variance. All of it.
When I dug a little deeper, I found that in 2022 the SMS message was delivering a revenue-per-email/sms-sent (RPE/S) that was 1,623% higher than the email messages.
Breaking it down further, the SMS generated the same average order value (AOV) as the email…
But a much higher conversion rate – 1,627% higher. Once again, the SMS message was the sole cause of the variance.
It all looked great, but also a little suspicious. That’s when I took a look at the cadence.
Do you see what they did? The original series sent the first effort 90 minutes after cart abandon. The second email was sent after 25 hours, and the third after 72 hours.
When they added the SMS message they kept those email cadences the same – and put the SMS text message first, 25 minutes after cart abandon.
That’s pretty aggressive. I know that I’ve been shopping online, gotten a phone call, and walked away from my cart for 25 minutes, with the plan to return and purchase when I’m off the phone. How about you?
My theory: I think the SMS text message is taking credit for revenue that would have been generated anyway, even without a text message being sent.
Here’s why. The charts below look at the YOY numbers for just the email messages; they exclude the SMS message.
| Top Left: Total revenue is down just 1% | Top Right: Revenue-per-email (RPE) is down 14%, a bit but not that much |
| Bottom Left: Average Order Value (AOV) is up, but just by 5% | Bottom Right: Conversion Rate from Sent (CR) is down 18%, again, a bit but not much |
As you can see from the start at the top left, there’s no significant difference in YOY email revenue from 2021. The YOY variance here is 1%. So the SMS message is not cannibalizing the email efforts. You would expect to see at least a bit of this when you are adding the SMS to the beginning. But it’s so early after abandonment that it’s having limited to no impact on the YOY performance of the email.
Then we look at RPE (top right). It’s down a little, 14%. This is very small compared to the magnitude of the SMS revenue increase.
AOV, at the bottom left, is also almost stable, just up by 5%. Again, that’s nothing, I’m not even sure it’s statistically significant.
Finally, at the bottom right, we have CR. It’s down a bit, 18%, but not much compared to how high the SMS rates are.
How can find out for sure what’s happening here?
We can take a step back and implement as it should have been done in the first place – with a holdout group.
I have lots of thoughts on the 25-minutes-after-cart-abandon timing. But those don’t really matter. If we set this up with a hold-out group, so that half the people who abandon their carts receive the first SMS text effort and half don’t, we’ll get a clean read on how much lift the SMS messages are actually providing. In other words, we’ll see what percentage of these sales the SMS is actually responsible for – and what percentage would have happened anyway, without the SMS text.
Why does it matter? Because the SMS text adds a cost – and if that fee is being paid on sales that would have happened anyway, then the client is losing money by adding this SMS text to its abandoned cart campaign.
Here are some potential scenarios…
Scenario A: The test (SMS) group generates significantly more revenue than the control (holdout) group
In this scenario, the SMS message is indeed generating the additional revenue reported. And with the level of revenue it is, it certainly covers the cost. So yes, the SMS message has significantly boosted revenue.
Scenario B. The test (SMS) group generates the same revenue as the control (holdout) group
Here the SMS message appears to be having no impact on revenue, as the revenue is the same with or without it. The company would actually be losing money on the SMS fees, since they would have gotten the sales without the SMS message.
Scenario C. The test (SMS) group generates a bit more revenue than the control (holdout) group, but not much more
In my mind, this is the most likely scenario. I do believe that the SMS message could be having a positive impact on revenue; the question is the magnitude of it. Is it enough to cover the SMS fees as well as the resources (mostly staff time) to set it up, report on it, etc. It may, it may not. That’s why we do the holdout group test.
And that’s why you should never make a material change to your email marketing program without testing into it first.
So if you’re thinking of adding SMS to one of your email campaigns, be sure to test with a holdout group before you roll it out. Give it a try and let me know how it goes. Looking for help with this or other email marketing challenges? Let’s chat and see if there’s a consulting engagement there!
Be safe, stay well,
