Statistical significance calculator

By the Gather team · Published August 18, 2026 · Last updated August 18, 2026 · Free, no signup, nothing stored

Two variants, two conversion rates, one question: is the difference real? This runs a two-proportion z-test and gives you the p-value, the confidence interval on the difference, and a plain-English verdict instead of a number you have to interpret.

What significance actually means

A significant result means the difference you observed is unlikely to have happened by chance alone, given your sample sizes. It does not mean the difference is large, or that it matters to your business.

The p-value is the probability of seeing a difference at least this big if there were genuinely no difference. A p-value of 0.03 means there is roughly a 3 percent chance of this result under the null hypothesis. Below your threshold, usually 0.05, you call it significant.

The formula

p_pooled = (cA + cB) / (nA + nB)
SE = sqrt( p_pooled x (1 - p_pooled) x (1/nA + 1/nB) )
z  = (pB - pA) / SE
p-value = two-tailed probability from the normal distribution

Four mistakes I see constantly

• Peeking. Checking the test daily and stopping when it goes green inflates your false positive rate badly. Set the sample size before you start and wait.
• Confusing significant with important. With a big enough sample, a 0.2 percent difference becomes significant. Ask whether you would act on it before you run the test.
• Testing many things at once. Run twenty comparisons at a 5 percent threshold and one will come back significant by luck. Correct for it or pre-register the one you care about.
• No hypothesis. If you cannot say in advance what you expect and why, you are mining, not testing.

Significance is not the same as knowing why

The test tells you B beat A. It never tells you what about B worked. That gap is why we run qualitative depth and quantitative scale in the same conversation: the respondent picks the winner, then the AI asks them why they picked it. You get the number and the reason together. More on the method in our research FAQ.

Frequently asked questions

What is statistical significance?

It means the observed difference between two groups is unlikely to be the result of random chance, given the sample sizes. Conventionally, a p-value below 0.05 is called significant.

What is a p-value?

The probability of observing a difference at least this large if there were truly no difference between the groups. Lower means less likely to be chance.

Does significant mean important?

No. With a large enough sample, tiny differences become statistically significant. Decide what size of difference would actually change your action before running the test.

Can I stop the test when it turns significant?

No. Checking repeatedly and stopping at the first significant result substantially inflates your false positive rate. Fix the sample size in advance.

Which test does this use?

A two-proportion z-test with pooled standard error, two-tailed. It is the standard test for comparing conversion rates between two independent groups.

Is this calculator free?

Yes. No signup, nothing stored, everything runs in your browser.