P-Value & Significance Simulator

A p-value is the most misread number in statistics. It is not the probability that the null hypothesis is true, and it is not the probability that a result happened "by chance". It is a probability about the data, computed in an imaginary world where the null hypothesis holds: how often would an experiment of this size produce a result at least this far from zero? This simulator lets you visit that imaginary world — and then come back and read one real experiment with clear eyes.

How it works

In the first view the null hypothesis is true by construction: every experiment samples from a population with no effect at all, and a one-sample t-test is run on each. Because the p-value is a probability computed under the null, when the null actually holds those p-values spread evenly over [0, 1] — no region is favoured — so about 5% of experiments land below 0.05 purely because the threshold sits at 5% of the range. That is not a flaw in the method; it is the definition of the threshold, made visible.

In the second view you choose the truth — an effect size in standard-deviation units, including zero — and one experiment is drawn and tested. Its t statistic and p-value are computed exactly from the Student-t distribution by the same core the site's self-test verifies against SciPy reference values; the simulation is used only where simulation belongs (the crossing rate of the design, i.e. its power, shown beside the normal-approximation formula). Every run is seeded (the mulberry32 generator), so the runs you see are reproducible — the same runs the site's tests verify. Two honest limits, stated plainly: crossing p < 0.05 says nothing about whether an effect is big enough to matter, and missing the line does not prove an effect absent.