Definition
Running an A/B test with too few recipients can produce misleading results. The required sample size depends on three factors: your current conversion rate (baseline), the smallest improvement you want to detect (minimum detectable effect), and your desired confidence level (typically 95%).
Quick Reference
| Baseline Rate | Min. Det. Effect | Sample Per Variant |
|---|---|---|
| 2% | 25% lift | ~15,000 |
| 2% | 50% lift | ~4,000 |
| 5% | 25% lift | ~5,500 |
| 5% | 50% lift | ~1,500 |
| 10% | 25% lift | ~2,500 |
| 10% | 50% lift | ~700 |
Why It Matters
Testing with insufficient sample size risks making decisions based on random noise. If your list is too small for a valid A/B test, run a sequential test over multiple campaigns or use a before-and-after comparison instead of a split test.
Related Glossary Terms
A/B Testing
A/B testing in email marketing is the practice of sending two variations of an email to a small sample of your list to determine which version performs better before sending the winner to the remaining subscribers.
Abandoned Cart Email
An abandoned cart email is an automated message sent to customers who added items to their online shopping cart but left without completing the purchase. It is one of the highest-converting email types in ecommerce.
AIDA Model for Email
The AIDA model (Attention, Interest, Desire, Action) is a classic copywriting framework used to structure email campaigns that guide subscribers from awareness to conversion.
AMP for Email
AMP for Email is a Google-developed framework that allows email messages to include interactive elements like forms, carousels, accordions, and live content. It turns static emails into dynamic, interactive experiences directly inside the inbox.
Anchoring Effect in Email Marketing
The anchoring effect is a cognitive bias where the first piece of information presented (the anchor) influences subsequent decisions, used in email to frame pricing and value perception.
Announcement Email
An announcement email is a dedicated campaign that communicates a specific update, milestone, or change to subscribers, from product launches and feature releases to company news and events.