Most email A/B tests fail before they start. A team picks a subject line that feels stronger, splits the list, and checks open rates the next morning. Whichever version leads gets rolled out, the revenue line barely moves, and nobody can say what was learned. The problem is not the testing itself. It is the order of decisions: the metric, the size and the stopping rule are usually chosen after the result is known.
A better test is planned in the reverse order. You start from the revenue question, choose the one change most likely to move it, work out how many recipients you need to detect a meaningful difference, and write down what result will count as a win. Only then do you hit send. This article walks through that protocol step by step, so that every test you run can be defended and reused.
An email A/B test is a controlled comparison of one change, sent to two randomly split, similar-sized groups, judged on one primary metric that was chosen before launch. When those conditions hold, a difference in results is far more likely to come from the change than from chance.
Start with the revenue question
Every test should answer a question that matters to the business. Opens and clicks are useful diagnostics, but they are steps on the way to revenue, not the destination. A change that lifts open rate by 20 percent while lowering revenue per recipient is a loss dressed up as a win, and a change that lowers opens by 5 percent while raising revenue per recipient by 15 percent is a clear improvement.
For that reason, choose the primary metric before anything else. For most programmes it is revenue per recipient or conversion rate. Name one guardrail metric as well, usually unsubscribe rate or complaint rate, so that a short-term gain cannot be bought with damage to the list.
Find the bottleneck before choosing a test
The next decision is where to test. Map your funnel from delivery to conversion and find the stage that loses the most people. Each stage responds to different levers. If opens are weak, the sender name, subject line and preheader deserve attention. If clicks are weak, the first screen, the offer and the layout are the likely problems. If clicks are healthy but purchases are not, the landing page and the path to checkout matter more than anything in the email.
Testing the bottleneck is more valuable than testing whatever is easiest to change. A ten percent improvement at a stage that loses ninety percent of people can outweigh a large gain at a stage you are already winning. Teams often spend months on subject lines when the real constraint sits in the offer.
Write the hypothesis and the test plan
A hypothesis is a sentence with a reason in it: "We believe a benefit-led subject line will raise click rate because our readers respond to savings more than to brand news." Writing the reason matters, because it forces you to say what you expect to learn whichever way the result goes.
The test plan around that hypothesis should record the variable, the audience, the primary and guardrail metrics, the sample size, the duration, the decision rule and what will happen if the result is inconclusive. Keeping it to one page is enough. The discipline is in writing it before launch, not in the length of the document.
Size the test so it can actually detect a difference
Sample size is the number of recipients each version needs before a difference of a given size becomes distinguishable from noise. Three inputs determine it: your baseline rate, the smallest lift that would change your decision, and the confidence level you require. Use a sample size calculator before launch and write the result into the plan.
This step exposes an uncomfortable truth about small lists. A list of 5,000 subscribers split in half gives each version 2,500 recipients. That is enough to detect a large change in a click rate near 5 percent, but not a subtle one. If your list is small, the honest response is to test bigger changes: a different offer, a different structure, a different reason to click. Wording tweaks on small lists produce mostly inconclusive results, which is an expensive way to learn nothing.
Run the test without interfering
Once the test is live, the main risk is your own impatience. A version that leads after two hours often falls behind once the full audience has opened, because early openers are not representative. Set the duration and sample size in advance, and do not judge a winner until both are reached. Repeatedly checking results and stopping the moment one version pulls ahead inflates the rate of false winners, sometimes dramatically.
Monitoring still has a place. Watch for delivery failures, broken links, rendering problems and unexpected spikes in complaints. Those are operational issues, and acting on them is part of running the test responsibly. Judging the winner is a separate step that waits for the end.
Call the result on revenue, then check the guardrails
When the test ends, begin with the primary metric. If one version wins clearly on revenue per recipient and the guardrails are stable, roll it out and record the decision. If it wins on clicks but revenue is flat, the lesson is that the next stage of the funnel is now the bottleneck, which is useful in its own right. If there is no significant difference, record that the change is not a major lever for this audience. If the revenue winner also lifts unsubscribes, investigate before rolling out, because the gain may be borrowing from future engagement.
A test that ends without a winner has still done its job if it tells the team where not to spend the next quarter.
Keep a test log that compounds
A single test teaches you one thing. A log of thirty tests, each with its hypothesis, audience, sample size and result, teaches you how your specific list behaves. After a year you will know which levers consistently matter, which ones never do, and which audiences respond differently. That knowledge is the real asset, and it only exists if every test is written down, including the failures.
For guidance on reading results, see email A/B testing reporting. For a candid look at why many tests are not worth running, read why most email A/B tests are a waste of time.
A/B testing pays off when it becomes a habit with a method behind it. The goal is not more tests. It is a repeatable process that turns each send into a decision you can defend.
Key Takeaways
- Every test starts with a written hypothesis and one primary metric tied to revenue or conversions.
- Find the bottleneck first: opens, clicks or conversions.
- Plan the sample size and test length before launch, then wait for the result.
- Judge winners on revenue per email, not on open rate alone.
- Implement winners quickly and log every result, win or lose.
Related Reading