The Story of Inference-Time Scaling and the Training-Time loop behind it

Bangalore Lakshman, Shashank; Eiers, Scott Lawrence

Act 2 · The Toolbox

Download the notebook and traces (zip, 2.8 MB) ↓

11. The Keystone: The Test-Time Compute Scaling Law

A bigger model and more thinking are two ways to spend the same FLOPs (floating-point operations, the raw count of arithmetic a chip does). Pretraining burns them once, into the model’s parameters; sampling and voting burn them at answer time, on every question you ask. Act 2 built the answer-time kind. Matched bill for bill on the deck’s own run, a 3B against a 7B, can thinking harder out-buy a bigger model?

1. The game, on our own instrumentbeat the bigger model’s one answer · measured, Qwen2.5-3B against 7B

Both roads leave the same dot: the small model’s own single greedy answer, ringed on the black line. Go up the black line and you are buying parameters, until the Qwen2.5-7B’s single greedy answer sets the bar at one times its own bill. Go right along the blue curve and you are buying samples and a vote instead. Push the samples and the blue curve climbs, but it never reaches the black line, here or several times past it.

a bigger model (scale up the parameters)the 3B, thinking harder (more samples + a vote)coverage: the answer was in the pile (the green risers are what the vote throws away)best-of-16 (a trained verifier)

both curves start on the small model’s own greedy dot, because at one answer there is nothing to vote over and nothing to have missed. measured on this deck’s instrument: the hard band is pages 6 to 8’s own 100 MATH-500 level 4 to 5 problems, and the run reproduces their shipped numbers exactly before it draws anything new (gate 1). FLOPs are counted 2 × parameters × tokens, from measured token counts.

Problem difficulty

Which model votes

hard problems, MATH-500 levels 4 and 5, n = 100

Test-time compute (samples the 3B votes over)

3B vote@16: 55%, still 5 points under the Qwen2.5-7B’s 60%

and now at 6.5× its bill. Even a trained verifier (page 8) only ties it, at 60%. No sample budget on this band crosses the line.

Why more samples do not cross: the bottleneck is not the thinking, it is the picking. The dotted green line is coverage: a correct answer is somewhere in the 16 sampled chains 78% of the time, above the Qwen2.5-7B’s 60%. The chains are not breaking; the reasoning is there. What fails is selection: voting returns that answer only 55% of the time. The answer is in the pile, and no rule we can run at answer time reliably grabs it.

2. The published law, and its fine printcited · Snell et al. 2024, on models from before the reasoning recipes

That was one model family, on one date. The published record says the opposite is possible: a small model that thinks can out-buy a bigger one. The rest of this page is that record, and the gap between it and what we measured. Start with the law itself. Snell et al. 2024 found that guessing how hard a prompt is first, then choosing the strategy and budget to match, can beat a bigger model at a matched bill. They call it compute-optimal.^{reference 1} Three numbers carry it.

4× fewer samples than best-of-Nbest-of-N spends 256 samples on every question, easy or hard. Guess the difficulty first and spend accordingly, and 64 samples buy the same accuracy

14× the parameters, matched on FLOPsgive both sides the same FLOPs bill: the 1× model buys 14 answers’ worth of thinking, the 14× model buys one greedy answer. The thinker scores higher, on problems it can already sometimes solve

R decides the regimeR = tokens you will serve ÷ tokens you trained on. Pretraining is bought once, and its cost is spread over every query you ever answer. Thinking is a surcharge on each one. Serve little (R → 0) and thinking wins; serve billions (R → ∞) and parameters win

Hard mode is the other boundary: where the small model has no foothold, no budget rescues it. Sampling can only ever surface an answer the model was already capable of writing. Drawing more samples turns up more of what it can write,^{reference 2} but it does not teach the model anything. If this model would never write the right answer, a thousand tries are a thousand copies of the same wrong idea. Parameters are the knob that changes what it can write at all.

3. Each scale owns a band of the budgetcited · Wu et al. 2024, Pythia 410M, 2.8B and 12B on GSM8K

The law is not only Snell’s claim; Wu et al. 2024 measured it. They took three real models on GSM8K (grade-school word problems), gave each steadily more samples, and had them vote, with each vote weighted by a scorer’s confidence. Sampling the small one many times beats sampling a big one a few times. And the crossings the law predicts show up in the data.^{reference 3} Section 2 allocated by prompt; this allocates by budget. Below, the budget axis is washed into one band per scale, the stretch of budgets where that model is the compute-optimal buy, and each curve runs bold through its own band.

the bar under each band name: where that handover could be

On a small budget the 410M model is the compute-optimal pick: sampling it many times beats sampling a big model a few times. Then every curve flattens. Once the 410M saturates near 60%, the 2.8B and then the 12B pull ahead, so each scale owns a band of the budget axis. The bars along the bottom say how firmly each handover is placed, and they are wide. Across each bar, the two leading curves differ by less than the ±3 points this figure was read to. So the crossing sits somewhere inside the bar, and not on the edge the wash has to draw. The second bar runs to the last budget where two models were still being compared, so the order of the handoffs is what this chart shows, not the budget at which the 12B takes over. This is a 2024-era family and a wide one, the 2.8B about 7× the 410M and the 12B about 29×, so the colours carry rank, not fixed multiples.

read off the paper’s figure by eye (±3 points) and replotted as accuracy; two of their five model sizes omitted for legibility. The washes tile the axis, but only the rail on top of each band was measured against a rival. Left of the 410M’s rail, the budget does not cover the 2.8B’s cheapest run. Right of the 12B’s, the 2.8B was not sampled any further. Out there the wash names the last model standing, not a winner over a rival

4. Better search moves the whole frontiercited · the same paper’s MATH500 table, sampling against tree search

Section 3 varied how much compute; this one varies how it is spent. The same paper’s results table on MATH500: three models, each under plain sampling and under their REBASE tree search, an arrow carrying each model’s pair from the gray point to the green one. Up and to the left is better.

sampling + weighted voting (64 to 256 samples)REBASE tree search + weighted voting (32 samples)

Wu et al. tested every inference strategy and both benchmarks. Across all of them they find the 7B model needs about half the total FLOPs of the 34B to reach comparable accuracy. This table’s sharpest instance is sharper still: 7B with tree search edges the 34B model’s best sampling run with roughly 8× fewer inference FLOPs.^{reference 3} The fine print holds here too: gains from more samples flatten toward a ceiling that the model’s own answer distribution sets, so without a perfect external verifier, sampling alone never reaches 100%.^{reference 4}

exact values from the paper’s MATH500 results table (Table 1); no digitization involved

So why did the crossing not appear for us? The law lives in a narrow window, where the model reaches the answer sometimes but not reliably, and on a 2026 post-trained model that window has shrunk toward the hardest problems. Two loose ends are left. The selection wall: the answer is in the pile, but no rule we can run at answer time reliably grabs it. And the waste: a chain that reached the answer is a worked solution the model wrote for itself, and the vote throws it away. Act 3 stops selecting at inference time and trains the reasoning into the weights. Page 12 keeps the winning chains and fine-tunes on them. Pages 13 and 14 reward only correct answers, until the reasoning emerges on its own.