5. Chain of Thought: Reasoning Out Loud

1. What a chain of thought isthe term, before the measurement

Page 1 handed you 17 × 24 and you did not recognise the answer, you built it: 340, then 68, then 408. A chain of thought is that building written down, a thread of steps running from the question to the goal.^{reference 1} What is strange in a language model is that the thread is not a report of the thinking, it is the thinking: every step is text the model has to write before it can write the next one. The classic way to get one is to ask, by appending Let's think step by step. to the question.^{reference 2}

2. What it looks likeone problem · one model · two prompts

A model you can download in 2026 does this natively. Nothing has to ask it to: given a hard maths question and nothing else, Qwen2.5-3B-Instruct writes 231 tokens of chain before it answers. A token is a word-piece, the unit a model writes one at a time, and it is what every length on this page is counted in.

Below is one MATH-500 problem put to it twice, on two prompts that are the same string apart from what is appended at the end. Both runs decode greedily: at every step the model takes its single most likely next token, so the same prompt always produces the same chain, and nothing below is a lucky draw. Pick a prompt and step through what it wrote.

You are a helpful math assistant.
Answer the question and write the final result on a new line as:
\boxed{ANSWER}

Question:
Half the value of $3x-9$ is $x+37$. What is the value of $x$?

Answer:

3. What the chain is worththe same 100 problems, both ways

One trace proves nothing, so both prompts were run across 100 problems from the same difficulty band. Below is that comparison: the same model, reasoning against non-reasoning. Reasoning, it writes a median of 573 tokens of working and solves 43 of the 100. Each square below is one of those 100 problems; the switch is the same suffix you just read.

4. The same test on two more checkpointsthree checkpoints · same 100 problems · same harness

Maybe the 3B is just an odd checkpoint, one saved set of a model's trained numbers among the many a lab ships. So the same test ran on its smaller and larger siblings, on the same 100 problems. Each bar below is one model's score, cut in two: on the left, what it still gets right with no reasoning; on the right, what the working adds. Bigger models are better at maths, so the bars get longer going down. Watch where the left piece ends instead.

The right-hand ends fan out with size, the way you would expect. The left-hand ends do not: non-reasoning, all three land in the same narrow strip, and that flat floor is the finding.

5. What each checkpoint wrote, and what it solvedthree checkpoints · both prompts

Figures 1 and 2 are one checkpoint, the 3B.^{reference 3} Here are all three, each plotted twice. Across is the median number of tokens the model wrote per problem, on a log scale because the two prompts sit two orders of magnitude apart. Up is the share of the same 100 MATH-500 problems^{reference 4} it solved.

Both numbers rise with the checkpoint, but not together. Reasoning, the three models write almost the same amount, a median of 561, 573, 582 tokens, and solve 22%, 43%, 60% of the same hundred. Non-reasoning, they write 5, 6, 6 tokens and solve 4%, 7%, 11%. The tokens move with the prompt; the accuracy moves with the model.