I always get a laugh out of research promising to reveal the ‘best day to send email.’ Because, think about it… if the research says ‘Thursday at 3:00 PM ET’ is the best day/time to send, and everyone started sending at 3:00 PM ET on Thursdays, then the best time to send would be any time but that.

Because at 3:00 PM ET on Thursdays everyone’s inbox would be bombarded with mail. It would be rush hour in the inbox and you’d be better off sending at another time when your email would have less competition for attention.

Also, the best day/time to send depends on who your audience is.

So how can you optimize your send day and time?

Don’t listen to the third-party research. Do your own research.

Case Study: A/B split test on best day/time to send email

Background

In this case study, the client had been sending on Mondays and Wednesdays, which had proven to be our best days to send to optimize revenue-per-email (RPE).

With the winter holidays approaching, we wanted to up our send frequency to three times a week – and we didn’t want to send two emails on the same day. So we were looking for which of the other weekdays, Tuesday, Thursday, or Friday, would be best to add.

Test Set-up

We didn’t technically have a ‘control’ here – since all of these were new send days. So we assigned Tuesday the moniker ‘control’ to make our reporting easier. This allows us to compare the other two days to Tuesday and see which of three does better.

Note: This is a B2B ecommerce company. We had tested sending on the weekends, and the results were not as good as sending on weekdays. So we did not consider weekend sends for this test.

In order to do testing like this, you have to make sure that the only difference between the sends is the day. Everything else must remain the same, absolutely everything, including:

• Send time
• Subject line, preheader text, and headline
• Body copy
• Call-to-action
• Landing pages
• EVERYTHING!

You also have to make sure you are segmenting the list in a random manner – that’s right. It’s not that you send the same email to the same people on different days. It’s that you segment your list into 3 random groups, with each group being mailed on one of the days. Like this:

My rule of thumb: make sure you have at least 20,000 email addresses per segment. This gives you your best chance of getting statistically significant results. In this case, the total list size was just under 500,000, so we had just over 150,000 per cell.

Which segment do you think won? Was Tuesday, Thursday, or Friday the best day for us to add a send?

I’ll give you a minute to think about it… then scroll down to get the answer.

Results

Here’s the data from the send…

Do your thoughts on the winner change when you see the data?

Our key performance indicator (KPI) here is revenue-per-thousand-emails-sent (RPME), because we are looking to optimize revenue.

Note: We used RPME instead of revenue-per-email (RPE) because the RPE numbers were small. By increasing the magnitude it makes it easier to see variances – and the relative variance between the cells remains the same.

The winner is Test 1 (Thursday), which delivered 59% more revenue than the control (Tuesday)

Test 2 (Friday) actually lagged the control (Tuesday), delivering 63% less revenue.

Let’s highlight the winners of each metric and further analyze this win.

So why did Test 1 (Thursday) do better?

RPME is determined by the conversion rate from emails sent (CR) and the average order value.

As you can see, both the Test 1 and the Control had conversion rates of 0.12%. No difference here. Side note: Test 2 had a much lower conversion rate, 0.005%.

Average order value (AOV) tells a different story. Here Test 2 bested the Control – it’s $424.84 AOV is 54% higher than the control’s $275.90 AOV. Once again, Test 2 lagged them both.

Thursday (Test 1), was the winner because people purchasing that day spent more money, on average, than people purchasing on a Tuesday (Control).  

It’s interesting to note that the usual metrics you think about with email – I call them diagnostic metrics – are all relatively the same. These are the open rate, click-through rate, and click-to-open rate.

Thursday (Test 1) did not win because more people opened or clicked on that day than they did on Tuesday (Control) or Friday (Test 2).

Take-aways

So, does this mean that you should shift your sends to Thursdays?

No!

It means that you should test different send days and times with your audience and find what works best for your situation.

Give it a shot and let me know how it goes!

Be safe, stay well,