Act 2 · The Toolbox
Download the notebook (zip, 2.0 MB) ↓6. Thinking Longer
Page 5 gave the model a chain. A chain has a length, and that length is compute you pay for, one token at a time. So put the length on a dial and turn it.
1. Cut it off, make it answerone problem · one model · the cap on a dial
The dial below is max_new_tokens, a hard limit on how much the model may write. Turn it down and the chain stops mid-thought, so at the cap we close its working off, start the answer line for it, and allow it 32 more tokens to finish.reference 1 That is why a starved model still produces an answer to grade rather than nothing at all. The tinted band in the trace below is that line, printed as we wrote it. Every figure on this page carries its own copy of this dial, so nothing you do here moves anything below.
Qwen2.5-3B-Instruct at bf16, greedy · MATH-500 level 4, Counting & Probability
Suppose that I have 6 different books, 2 of which are math books. In how many ways can I stack my 6 books on a shelf if I do not want the math books to be next to each other?
This chain runs to 351 tokens and stops on its own, so the dial bites at 128 and 256 and then has nothing left to cut: 512 and up are the same chain.
the model →
budget spent · we close the block and make it commit
Final answer: 360
or run a different problem · 8 of these ship with their full traces
The grey text is what the model would have written and was not allowed to; the tinted line is ours, not its conclusion. Step through the stops and the verdict does not slide, it flips: for one problem, thinking longer is a threshold and not a gradient. That is Act 1's slow deliberate System 2, priced in the only currency a model has.
2. The same dial, every problemthree checkpoints · the same 100 problems
One problem is an anecdote. Below is the same sweep run over all 100, on three checkpoints: how many each solves at every stop on the dial. The line on the chart is this figure's own dial, so drag it. Read the shape rather than the heights.
at 128 tokens, 3B-Instruct solves
7 of 100
tokens generated to get there
13,211
the cap bound 100% of these 100 chains
the procedure · Cap generation at N tokens; if the model is still inside its think block, close it and append "Final answer: \boxed{"; allow 32 more tokens; grade with MATH-500's reference answer. Repeat for N in {128, 256, 512, 1024, 2048} over 100 problems from levels 4-5, greedy throughout.
Every curve climbs steeply and then enters the same slab: past 1024 tokens, one more doubling moves the 3B down by 1 point, which is one problem out of 100. Three heights, one shape, and the shape is the part that is not about any of these checkpoints.
3. It stops mattering before you run out of dialone square per problem · placed by how long its chain ran
So why does the curve flatten? Below are the same 100 problems again, each one placed by how long its chain ran when nothing cut it off. The line is the budget you are holding: everything to its left finished on its own, everything to its right was still being cut, and here the line is the control: drag it along the axis and watch it move past the pile.
and where each problem first grades correct
At 2048 tokens the shaded strip holds 6 squares. Every other chain stopped when it had nothing left to say, a median of 655 tokens for the ones that never come right, with 1393 still unspent. So 46 of these problems did not run out of room, they ran out of ideas. The dial was never what was holding them back.
4. The other direction: refuse to let it stopthe same 100 problems · not allowed to stop
Everything above takes room away. This does the opposite: when the model tries to end its turn, the stop is dropped, the word Wait is appended to what it has written, and it carries on. 3 times over.
9 answers turned from wrong to right against 10 that turned from right to wrong, p = 1.000. Both crossings are thick and they cancel. In all, 30% of answers moved, because a further 11 swapped one wrong answer for another and never touched the verdict.
and the same crossing on all three checkpoints
| checkpoint | right → wrong | wrong → right | solved, and what it cost | |
|---|---|---|---|---|
| 3B | 10 | 9 | 43% → 42% · 30% of answers moved · ×1.8 the tokens · p = 1.000 | |
| 1.5B | 9 | 7 | 22% → 20% · 59% of answers moved · ×2.2 the tokens · p = 0.804 | |
| 7B | 7 | 4 | 60% → 57% · 21% of answers moved · ×1.6 the tokens · p = 0.549 |
Both counts are against the same problem at zero injections, so they are paired.
It redistributes rather than adds. Spending 1.6 to 2.2 times the tokens buys a third of the answers changed and no more of them right: across the three checkpoints it turns 20 wrong answers right and 26 right answers wrong. Not one of the three crossings resolves, so the direction is a null and not a finding: what is measured here is that a great deal moved and the totals did not.
5. Where the length runs outthe same 2,048 tokens · two ways to spend them
The dial flattened at 43%, and 51% of these problems never came right at any length. But a budget is not a length: the same 2,048 tokens can be one long chain or 16 short ones, and page 7 turns on the difference.