What does it take for an LLM to be useful (helpful, harmless and honest)reference 2 in the real world? What are the pitfalls of System 1 thinking in the real world?
1. Answering a question
How many FIFA World Cups has Spain won?
the model answers
✗ one short
“Spain has won the FIFA World Cup once, in 2010.”
A model learns everything it knows before a fixed training cutoff, and the answer it gives depends on when it was trained. This creates a perpetual knowledge gap after the model is trained, as new data is created by the world.
Based on the knowledge cutoff date, the model gives an answer without thinking it through. System 1 thinking leads to an incorrect response.
Are knowledge gaps addressable by System 2 thinking?
2. Answering a question without bias
The question never said men's. Spain fields a team in both tournaments, and has won three FIFA World Cups between them:
the ideal answer
✓ all 3 titles
“Spain has won three FIFA World Cups: the men's team in 2010 and 2026, and the women's team in 2023.”
Every answer above quietly counted only the men's tournament. That is a knowledge bias, and it is not the knowledge gap.
The 2023 win happened before the cutoff, so the model held the fact the whole time and left it out anyway. Nothing was missing from its memory. Handed an ambiguous question, it resolved the ambiguity to whichever reading its training data talked about most, and never mentioned that it had chosen. That is what makes a knowledge bias different from a knowledge gap: the gap is about what the model was never shown, the bias about which part of what it holds it decides to use. Models do this in the small too: given a sentence that could refer to either of two people, they resolve it to the stereotype-consistent reading rather than flagging that the sentence is ambiguous.reference 1
Are knowledge biases addressable by System 2 thinking?
3. Thinking it through
Both misses share a shape: the model answered immediately (System 1 thinking). It never reread the question, and it never weighed what it might not know. A System 2 pass does both before committing to anything.
Step #1Read the question againIt says FIFA World Cups, not men's World Cups. Spain fields a team in both tournaments, so both could count.
Step #2Check my own limitsA count of titles changes whenever a final is played, and my training was sealed in early 2025. Any final after that falls outside what I know.
Step #3Do not answer from memory aloneSay which tournaments I am counting, and check a current source before committing to a number.
Step #4Count what the question actually askedThe men's team won in 2010 and 2026, and the women's team won in 2023.
Look at what each step bought. Step 1 settles the bias outright, and it needed no new information: the ambiguity was sitting in the question the whole time, and reading it carefully was enough. Step 2 does not close the knowledge gap, because thinking cannot conjure a fact the model was never shown. What it does is notice the boundary, which turns a confident wrong answer into a flagged one, and a flagged one into a reason to go look.
Nothing about the model changed between the reflex and the trace. Same weights, same training data, same cutoff. The only difference is that it spent effort before answering. That effort is the subject of the rest of this story: what it buys, how to buy more of it, and where buying more stops helping.