A/B Test Sample Size Calculator

It's the smallest real difference between your control and variant you actually want the test to be able to detect. Setting it too small (chasing tiny improvements) demands a huge sample size, while setting it too large means the test might miss a genuinely meaningful but smaller improvement. Absolute mode adds percentage points directly to the baseline (10% + 2pp = 12%); relative mode scales the baseline by a percentage (10% x 1.20 = 12%).

Confidence level controls your tolerance for a false positive, declaring a winner when there really is no difference (a 95% confidence level means a 5% chance of that happening). Statistical power controls your tolerance for a false negative, missing a real difference that's actually there (80% power means an 80% chance of correctly detecting the effect if it exists, a 20% chance of missing it).

Every extra comparison against the control is another chance for a false positive to slip through, so testing more variants at the same per-comparison confidence level quietly raises your true overall error rate. The Bonferroni correction tightens the confidence level for each individual comparison so the combined risk across all of them still matches your originally chosen confidence level, at the cost of needing a larger sample size per variant.

Detecting small effects reliably is inherently expensive in terms of sample size, the required size grows roughly with the inverse square of the effect you're trying to detect, so cutting the MDE in half roughly quadruples the sample size needed. This is a real statistical constraint, not a flaw in the calculation, it's the same tradeoff every A/B testing platform faces.

It divides the total sample size needed across all variants by your daily visitor count (split however your traffic is actually divided among variants) and rounds up to the nearest whole day. It's a planning estimate assuming steady, consistent traffic, actual test duration can run longer if traffic dips or if you also want to account for weekly cycles in user behavior.

It assumes a standard two-proportion z-test comparing conversion rates, the most common approach for typical A/B tests measuring a binary outcome like "converted or didn't." If you're measuring a continuous metric instead, like revenue or time on page, a different sample size formula based on that metric's variance would apply.