# A/B Test Significance Calculator

Check if your A/B test results are statistically significant. Two-proportion z-test with p-value, confidence intervals, and relative lift calculations.

## What this calculates

Determine if the difference between your A/B test variations is statistically significant. Enter visitors and conversions for each group to get a p-value, relative lift, and confidence interval.

## Inputs

- **Control Visitors** — min 1 — Number of visitors in the control group (A).
- **Control Conversions** — min 0 — Number of conversions in the control group.
- **Variation Visitors** — min 1 — Number of visitors in the variation group (B).
- **Variation Conversions** — min 0 — Number of conversions in the variation group.
- **Significance Level** — options: 90% Confidence (α = 0.10), 95% Confidence (α = 0.05), 99% Confidence (α = 0.01) — The significance level for the hypothesis test.

## Outputs

- **Control Conversion Rate** — Conversion rate of the control group.
- **Variation Conversion Rate** — Conversion rate of the variation group.
- **Relative Lift** — Percentage improvement of variation over control.
- **P-Value** — Two-tailed p-value from the z-test.
- **Statistically Significant?** — formatted as text — Whether the result is statistically significant at the chosen level.
- **Difference CI** — formatted as text — Confidence interval for the difference in conversion rates.

## Details

A/B testing compares two versions (control A and variation B) to determine which performs better. Statistical significance tells you whether the observed difference is real or likely due to random chance.

The method: This calculator uses a two-proportion z-test. It computes a pooled proportion, calculates the standard error, and derives a z-statistic. The p-value represents the probability of observing a difference this large (or larger) if the two versions truly had equal conversion rates.

Interpreting results: A p-value below your significance level (typically 0.05) means the result is statistically significant - the variation likely performs differently from the control. The confidence interval shows the plausible range of the true difference. Always ensure adequate sample size before drawing conclusions; premature testing inflates false positive rates.

## Frequently Asked Questions

**Q: How many visitors do I need for a valid A/B test?**

A: It depends on your baseline conversion rate, the minimum detectable effect, and desired confidence level. As a rough guide, for a 5% baseline rate and 20% relative lift detection at 95% confidence with 80% power, you need approximately 4,000 visitors per group. Use a power analysis calculator for precise estimates.

**Q: What does a p-value of 0.05 actually mean?**

A: A p-value of 0.05 means that if the two variations truly had identical conversion rates, there would be a 5% chance of observing a difference as large as (or larger than) what you measured. It does NOT mean there is a 5% chance your result is wrong.

**Q: Can I stop my A/B test early if results are significant?**

A: Stopping early (peeking) inflates false positive rates because significance can fluctuate. If you plan to check results multiple times, use sequential testing methods (like O'Brien-Fleming boundaries) or wait until the planned sample size is reached.

**Q: What is relative lift and why does it matter?**

A: Relative lift is the percentage improvement: (variation rate - control rate) / control rate x 100. A lift of 20% on a 5% baseline means the variation converts at 6%. Lift tells you the practical significance, which is separate from statistical significance.

---

Source: https://vastcalc.com/calculators/statistics/ab-test
Category: Statistics
Last updated: 2026-04-21