Act 3 · Frontier & Caveats

the runs behind the chart:page 5 (0.8 MB) ↓page 6 (2.0 MB) ↓page 7 (3.0 MB) ↓page 8 (9.7 MB) ↓page 9 (1.6 MB) ↓

16. Wrap-Up: What You Now Know

The past fifteen pages are a small but mighty snapshot of what inference-time scaling is under the hood. Reasoning turned out to be one knob that can be turned many different ways, and that can even be moved from one axis to another. Here is every lever on the first axis, measured on the same problems the pages showed you. Then ten questions, to see whether the mechanisms landed rather than whether the numbers stuck.

1. The whole toolbox, on one axisour runs · 100 problems · pages 5 to 11

Each page measured its own lever against its own baseline, so none of them could draw this. Now that all of them are behind you, they go on one axis, priced in the compute they actually spent.

0255075100no chain at all, forced to answer immediately · page 57%one chain of thought, greedy · page 543%one chain, forced to keep thinking · page 642%16 chains at temperature zero, voted · page 743%16 sampled chains, voted · page 755%16 sampled chains, trained verifier picks one · page 860%sampled chains, filtered by step soundness · page 957%58%at least one of the 16 was right · page 778%accuracy on 100 MATH-500 level 4 and 5 problems

Read it top to bottom as steps that keep getting smaller. Writing the reasoning at all is the largest single move on the chart, and being made to write more of it is the only step that goes backwards.

Every chain above was written by Qwen2.5-3B. One model wrote all of it. What changes down the chart is how many chains it was allowed to write and what got to pick between them, never who was writing; where a judge appears it scores chains, it does not write them. The run behind each rung is linked at the top of this page, traces included, so any of them can be re-derived without a GPU.

1b. Conclusionssix things the chart is evidence for

1.Letting the model reason wins, every time

Ask for the answer on its own and it mostly gets it wrong. Let it write its working first, on the same problems, and it gets far more of them right. But making it write more working, when it has nothing new to go on, is a different thing: it costs a great deal more and turns about as many answers wrong as it turns right. Reasoning is worth having. It is not worth forcing.

2.The answer is usually already in the pile

Ask the same question many times and a right answer turns up far more often than any rule we tried managed to pick out. The gap between what was found and what was chosen is not a thinking problem, and sampling harder does not close it.

3.So the judge is the bottleneck, and judges can barely read

Something has to choose. An arithmetic checker can only mark the small part of a chain that is arithmetic, and the rest is prose it cannot read at all. Meanwhile the more variety you buy by sampling, the worse every model asked to judge gets, while a model trained to score holds steady. So you want a trained judge. But sampling and judging your way to what a bigger model reaches on its own costs many times more compute than simply using the bigger model.

4.Adaptive compute is where that points, not where it lands

Spend more on the hard questions and less on the easy ones, rather than the same on all of them. That is the obvious next move, and this deck could not make it pay: even picking the best stopping point for each question with hindsight bought no more per unit of compute than plain resampling. Knowing where to spend only helps if you can know it in advance.

5.And you pay for every node you look at

Branching through partial chains and cutting the bad ones sounds cheaper than it is. You pay for a branch before you can judge it, so pruning never refunds what looking cost. It has no bar here because this deck never ran it, and running it would mean holding a second, larger model in memory beside the one doing the writing.

6.And the biggest move in this story is not on this axis

The largest jump in the whole deck came from none of the levers on this chart. It came from taking a small model and training it on the reasoning of a much stronger one. It is not a bar here because it was not the same test: the published result covers every difficulty level and this chart is only the hardest ones, and our own models score noticeably higher on the easier mix. So part of that gap is the easier test and part is the training, and this deck cannot tell you how it splits.

2. Ten questionsconcepts, not numbers

In the deck's order. Each answers itself as soon as you pick, and says which page it came from. The options are shuffled on every visit, so where an answer sat last time tells you nothing. Nothing here needs a number memorised: where a figure appears it is the setting the question is asked in, not the thing being asked.

0 right of 0 answered, 10 in all

1 of 10page 2 · what thinking longer can reach

A model gets one question wrong in two separate ways. It quotes a record that changed after its training data was collected, and it assumes the question is about the men's tournament when the question never said so. You hand it back and let it think the question over again, with nothing new to go on. Which mistake can it fix?

2 of 10page 3 · what retraining cannot buy

Now imagine retraining were free. No compute bill, no shortage of fresh text, no waiting weeks for a run to finish. You rebuild the model from scratch every morning. Which of those two mistakes is still there tomorrow?

3 of 10page 5 · what a chain of thought is

A model writes out its working, then gives an answer. To the model, what is that written working?

4 of 10page 6 · reading a flat curve

You raise the token budget and accuracy climbs, 44% at 1024 tokens, then it goes flat: 43% at 2048. One more thing to know. 94% of these chains stop themselves before the biggest budget. What is the flat part telling you?

5 of 10page 7 · what voting needs

Sixteen chains sampled at temperature zero scored 43%. One chain on its own scored exactly the same. Why did the other fifteen buy nothing?

6 of 10page 7 · coverage against accuracy

Sample 16 chains hot. The majority vote gets 55% of the problems right. But on 78% of them, at least one of those chains had the right answer. What is that gap made of?

7 of 10page 9 · why the answer is not enough

A model works through a problem, slips on one division halfway down, and still lands on the right final answer. Two graders look at it. One compares only the final answer to the known one. The other reads every line. What does the first grader say?

8 of 10page 10 · what search costs

Tree search branches into several partial lines of reasoning, scores them, drops the weak ones and backs up to try somewhere else. What does the compute bill scale with?

9 of 10page 12 · what self-improvement rests on

STaR trains a model on reasoning it wrote itself. R1 was taught to reason by rewarding it for getting answers right. Strip both loops back to the one thing neither works without. What is it?

10 of 10page 15 · why unaided revision fails

Ask a model to look over its own answer and fix any mistakes. No hints, no tools, nothing it did not already have. On CommonSenseQA that took GPT-3.5 from 75.8% down to 38.1%. Why does asking make it worse?

The one sentence to leave with: a model can get dramatically better answers by thinking longer at answer time, the gains are real and measured and each one has a price and a ceiling, and every one of them is paid in text that nobody has yet shown is the reason for the answer. That is where the field is, and where you now are.