Email A/B Test Sample Size Calculator
Determine how many recipients you need per variant to get statistically significant A/B test results. Plan your test before you send.
Configure Your Test Parameters
Your current conversion rate (e.g. 5 means 5%)
Relative lift you want to detect (e.g. 10 means 10% lift)
Confidence required to declare a winner (default 95%)
Probability of detecting an effect if it exists (default 80%)
Sample Size Results
Required Sample Size Per Variant
—
Recipients needed in each group
Total Sample Size Needed
—
Both variants combined
Minimum List Size Needed
—
Assuming 35% open rate for test eligibility
Confidence Level Benchmarks
90% Confidence
Z = 1.645
Low confidence, exploratory tests
95% Confidence
Z = 1.96
Standard for most A/B tests
99% Confidence
Z = 2.576
High confidence, critical decisions
Higher confidence levels require larger sample sizes. Use 90% for low-risk exploratory tests and 99% for high-stakes decisions where false positives would be costly.
How Sample Size is Calculated
This calculator uses the standard formula for two-proportion z-test sample size:
p₁ = baseline_rate / 100 (control proportion)
p₂ = p₁ × (1 + MDE / 100) (treatment proportion)
p̄ = (p₁ + p₂) / 2 (pooled proportion)
Zα: 90% → 1.645, 95% → 1.96, 99% → 2.576
Zβ: 80% → 0.842, 90% → 1.282
Sample Size Per Variant = ((Zα + Zβ)² × 2 × p̄ × (1 − p̄)) / (p₂ − p₁)²
Total Sample Size = Sample Size Per Variant × 2
Minimum List Size = Total Sample Size ÷ 0.35 (assuming 35% of your list opens the email and enters the test)
Results are rounded up to the nearest whole number. Always round up to ensure your test is properly powered.
Why Sample Size Matters for Email A/B Testing
Running an A/B test with too few recipients is one of the most common — and most costly — mistakes in email marketing. When your sample size is too small, your test lacks statistical power, meaning even if there is a real difference between versions, your test is unlikely to detect it. The result? You might conclude there's no winner and stick with the status quo, missing out on genuine improvements to your open rates, click rates, and conversions.
Conversely, a sample size that is too large wastes your list's potential. Every person in a test who receives a losing variant is a missed opportunity. The goal is to use the smallest sample size that is large enough to reliably detect the effect you care about. This is where a sample size calculator becomes essential — it helps you find the sweet spot between statistical rigor and practical efficiency.
Baseline Rate and MDE: The Two Key Drivers
Your baseline conversion rate has a huge impact on required sample sizes. Low baseline rates (e.g. 2% click rate) need much larger samples than high baseline rates (e.g. 20% open rate). The Minimum Detectable Effect (MDE) also matters: detecting a tiny 5% relative lift requires a massive sample, while a 20% lift can be detected with far fewer recipients. Choose an MDE that represents a meaningful business improvement, not just any statistically significant difference.
Power and Significance: Balancing Risk
Statistical power (typically 80%) is the probability that your test will detect a real effect if one exists. Lower power means you risk false negatives — missing a winning variant. The significance level (typically 95%) controls false positives — declaring a winner that isn't real. Both require larger samples as you tighten them. For most email tests, 80% power and 95% significance strike the right balance. Use 90% power only when the cost of missing a winner is very high.