What a p-value Says
What is the probability actually about?
The idea
A p-value gives the probability of seeing data this extreme if there were no real effect, not the probability that the effect is real.
In the real world
p = 0.04 does not mean a 96% chance the finding is true.
Going deeper
A p-value is computed assuming there is no real effect, and it measures how unusual the observed data would be under that assumption. It does not measure the probability that the assumption is false.
Reading p = 0.03 as a 97% chance the finding is true inverts the conditional. Whether the effect is real also depends on how plausible it was beforehand and how many things were tested, neither of which enters the calculation. This is why implausible findings at p < 0.05 replicate poorly, and why the number is a weak summary of evidence on its own.
Where it stops applying
p-values are still useful as one input, particularly with pre-registered hypotheses and adequate samples. The problem is treating a threshold as a verdict.
Why it matters
The misreading is extremely common and reverses what the number actually claims.
Try this today
When you see a p-value, restate it as if there were no effect, how surprising is this data.
Test yourself
A result is reported at p = 0.03 and described as 97% likely to be true. Why is that wrong?
Show the answer
The p-value is calculated assuming no effect exists and measures how unusual the data would be under that assumption. It says nothing about how likely the assumption itself is, which also depends on how plausible the effect was beforehand and how many things were tested.
Learn this in the feed Answering from memory, then again days later, is what makes it stick.