Hypothesis testing

Hypothesis testing is built around a deliberately asymmetric setup. You assume the null hypothesis — usually the boring, no-effect claim — is true, and ask: if it really were true, how surprising would this sample be? A test statistic, like z = (x̄ − μ₀)/(s/√n) for a sample mean, converts that question into a single number measuring how many standard errors the sample sits from what the null hypothesis predicted.

The p-value is the probability of seeing a test statistic at least this extreme if the null hypothesis were true — not the probability that the null hypothesis is true, which is a different question the data alone cannot answer. When the p-value falls below a chosen significance level (commonly 0.05), the result is called statistically significant and the null is rejected; otherwise, the honest conclusion is that the data didn't provide enough evidence against it — never that the null has been proven, since absence of evidence against a claim is not evidence for it.

Choosing the significance level is itself a trade-off, not a dial to turn toward "more rigorous" without cost. A smaller significance level makes a false positive (rejecting a true null — a Type I error) less likely, but only by making a false negative (failing to detect a real effect — a Type II error) more likely. Good test design balances both risks against what a wrong conclusion would actually cost.