Small Samples Lie
How many results does it take before a result means something?
The idea
Small samples produce extreme results by chance far more often than large ones.
In the real world
A landing page at 3 conversions from 40 visits looks decisive and is not.
Going deeper
Small samples produce extreme results by chance far more often than large ones, so a striking ratio from few observations is weak evidence rather than strong.
Percentages are what conceal this. Three conversions against six sounds like twice as good; stated as three events against six events, the fragility is obvious, since a swing of two or three is well within ordinary variation. The habit worth building is to look at the counts before the ratio, because the ratio is designed to hide how few things actually happened.
Where it stops applying
Waiting for statistical certainty on every decision is its own failure. Small samples are often all you have, and the right response is to hold the conclusion loosely rather than to refuse to act.
Why it matters
It stops you shipping a decision built on a difference that would vanish with more data.
Try this today
Before acting on a test, ask how likely the gap is with the number of observations you have.
Test yourself
Variant A converts 3 of 40 and variant B converts 6 of 40. B looks twice as good. Why is that not yet a finding?
Show the answer
With numbers this small, a swing of three conversions is well within ordinary chance, so the same test rerun could easily reverse. Ratios are stable in large samples and wild in small ones, and the percentage difference conceals how few events produced it.
Learn this in the feed Answering from memory, then again days later, is what makes it stick.