Email Testing Best Practice : How to Run an A/B Test

Before you begin to run your email testing, you'll want to make sure that you understand the best way to split your email list for a clean A/B test. Because you want the resulting lists to be as even as possible in terms of valid email addresses and demographic information, consider taking the following steps before splitting your email list into two parts:

Remove Inactive Users: Inactive users in your email list are useless. These are the ones who have never opened an email, or it may be users who haven't opened an email in a very long time. You'll need to decide for yourself what the criteria will be.

Remove Highly Active Users: Also remove your most active users before sending testing emails. This most likely will mean people who open the majority of the emails that you send, but it may also simply mean anybody who has opened an email within the last month or the last two email sends.

Sort Alphabetically: The best way to sort your email list is the alphabetical order. Then split it down the middle. It will often give you the most randomized data.

Do Not Sort by Join Date: However, if you choose to sort your list before splitting it, be sure that the list is not sorted by the sign-up or join date of the users. Your most recent sign-ups will be more active and likely to open than your older ones. If you sort by join date, one list is likely to respond to the email better than the other one based simply on the fact that they have engaged with your company more recently.

Split Your Highly Active or Highly Inactive Users Separately : After you have sorted and separated your main list, use the same technique to sort and separate your highly active or highly inactive users. Then split the results back into the two new segments of your main list. This will ensure that when you send testing emails, each of your A/B segments will have a sampling of average, highly active, and highly inactive users.

Check the Percentage of Email Service Provider Addresses: Do a quick check of each of your newly halved email lists. Make sure that you have roughly the same percentage of Hotmail, Gmail, Yahoo! and any other large email service providers on each list. If you do not, you may risk having your results altered if one of your email service providers sends your message to the junk or spam folder.

More Tips


All-in-One Marketing Automation Software