The Story of Inference-Time Scaling and the Training-Time loop behind it

Bangalore Lakshman, Shashank; Eiers, Scott Lawrence

Act 3 · Frontier & Caveats

14. Frontier Reasoners II: DeepSeek-R1, GRPO, and the “Aha Moment”

Page 13 closed with four questions about what sits behind the thinking dial, and no way to answer any of them from outside. In January 2025, DeepSeek published all of it: algorithm, weights, training curve, and a chain that stops mid-solution to re-check its own work. Where does the backtracking come from?^{reference 1}

1. The recipe: the group grades itselfGRPO · rule-based reward · no worked examples

R1-Zero deletes the usual supervised warm-up entirely and starts RL from a plain base model, with no worked examples at all: only problems whose final answers can be checked. The whole recipe, Group Relative Policy Optimization, fits in five lines:^{reference 2}

for each problem q:

sample G answers from the current policy

rᵢ ← 1 if answerᵢ is right, else 0# the reward is a program, not a second model

Aᵢ ← (rᵢ − mean(r)) / std(r)# the group is its own baseline: no critic network

push toward answers with Aᵢ > 0, away from Aᵢ < 0# plus a leash (KL) back to the reference model

no critic network anywhere, which at 671B parameters is what made RL at this scale affordable: PPO, the standard alternative, learns its baseline from a second network about as large as the policy itself

Line 2 should look familiar. Page 7 sampled many chains and picked among them at answer time, then threw every sample away once the vote was counted. GRPO runs the same sampling during training and keeps what the winners did in the weights: sampling paid for once, instead of on every question.

each sample's push is its distance from the dashed rule, its reward minus the group mean (GRPO then divides by the group's spread)

of the other three samples, how many are also right

Same right answer, luckier group: sample 1's push drops to +0.50.

2. Scrub through R1-Zero's trainingAIME 2024 · pass@1 · DeepSeek-AI 2025

Nothing in the recipe mentions reflection. Nothing in the reward mentions length. Watch both curves anyway.

RL training step

15.6%

AIME 2024 pass@1

≈1,000

avg response tokens

From 15.6% to 71.0% pass@1 with no supervised examples. Add page 7's majority vote across 64 samples and it reaches 86.7%, level with OpenAI-o1-0912.^{reference 1}

Length is not a schedule anyone set: longer, re-checking traces win more reward, so the model buys itself more thinking time. The skeleton below shows what the extra tokens are made of.

the trace, as a shape · early in training

≈1,000 tokens

A few steps forward, an answer, stop. Nothing looks back.

Count the traces: there is one, allowed to run longer. Not many chains sampled and voted on behind an API, which is the distinction page 13 could only assert.

Notice what the loops rule out. A chain nobody can read might be a running commentary written beside the real work, and from outside there is no way to tell. These tokens cannot be that: the chain is what the model sampled and what training pushed on, and no separate step ever wrote a description of anything. Re-derivation is not narration, and the score climbs as the loops arrive.

the skeleton abstracts the trace in Table 3 of DeepSeek-AI 2025, where R1-Zero interrupts a solution mid-stream, flags the moment, and restarts from its own first line at “an intermediate version” of training; the structure is theirs, no text is quoted, and where the aha lands on the slider and the chart is an illustrative placement: the paper gives the behaviour, not a step number

3. Then it was given awayTable 5 · ~800K samples · MIT license

Page 13's two knobs were never equally available. Anyone can pay for thinking at answer time; lifting the curve took frontier-scale RL, which meant being a frontier lab. Then R1 wrote out ~800K of its own reasoning traces, six small models were fine-tuned on them, and the behaviour came with the text.^{reference 1}

The curve-lifting knob got cheap. Not free: someone still paid for the RL once. But they paid once, and the result copies, so the reasoning does not have to be re-earned by anyone who wants it. The claim to check on the chart is not “R1 is good”; it is that a lifted curve fits in weights you can download. Pick a size:

Qwen baseLlama baseR1 teacher (671B MoE)closed model (size undisclosed)

plain supervised fine-tuning on the traces, no RL and no reward, which is page 12's recipe with a far stronger teacher · the released R1 adds a small cold start of curated traces before its own RL, and scores 79.8% here · rows are ordered by size, smallest at the top; the teacher's 671B is a mixture-of-experts total that activates ~37B per token · GPT-4o and o1-mini publish scores but not sizes, so they are rules, not rows

R1-Distill-Qwen-14B · ~48× smaller than its teacher, clears the best open reasoning model that existed the week before, at under half its size.

Where a dot lands right of a dashed rule, a model you can download beats a closed frontier product by thinking longer at answer time. And page 3 said the stock of human text was finite: these traces were not in it. One good reasoner now writes the curriculum for everyone else.

Two pages of frontier claims, and every one of them arrived as a benchmark score: 71.0% on AIME, level with o1-0912, a 14B crossing o1-mini's line. Every one of them is also a number for thinking longer. Next: the side of that dial where thinking longer makes the answer worse.

Simulated: the training curve's intermediate points are interpolated along the reported 15.6% → 71.0% climb; the trace skeleton is a schematic of the Table 3 trace's structure, not its text; and where the aha lands on the slider is an illustrative placement; the paper gives the behaviour, not a step number.

Cited: every benchmark number is transcribed from DeepSeek-AI 2025 (Tables 3 to 5, Figures 2 and 3) and pinned in the test suite.