AI Is a Probabilistic Tool
Why does wrong output from a language model usually read better than correct output from a search?
The idea
Generative AI predicts useful outputs; it does not guarantee truth.
In the real world
A draft can be fluent while containing a fabricated citation.
Going deeper
A language model is trained to produce text that plausibly continues what came before. Accuracy correlates with plausibility across most of its training data, which is why the output is usually right — but the objective is plausibility, and where the two diverge, plausibility wins.
This has a specific consequence for where you check. Errors do not arrive flagged as uncertain; a fabricated citation is generated with exactly the same confidence as a real one, because a plausible-looking citation is precisely what the objective rewards. Verification therefore has to be attached to the category of claim — names, numbers, references, quotations — rather than triggered by the output seeming doubtful.
Where it stops applying
This is not an argument that the output is unreliable in general. For tasks where being approximately right is fine, or where you can check the result yourself, the probabilistic nature costs you very little.
Why it matters
It tells you where to place a verification step: on the claims a model produces most confidently, not the ones that look uncertain.
Try this today
Add a verification step for consequential AI output.
Test yourself
A generated report is fluent, well structured, and cites a source that does not exist. Why did the fluency not warn you?
Show the answer
The model optimises for plausible continuation, not truth, and a fabricated citation is highly plausible text. Fluency is evidence the output is well-formed, and no evidence at all that it is accurate.
Learn this in the feed Answering from memory, then again days later, is what makes it stick.