Act 3 · Frontier & Caveats

12. Teaching Models to Reason: STaR and Self-Improvement

A model writes its own reasoning, an answer key discards what's wrong, and the survivors become its next training set. What happens when you take the answer key away?

1. Run the loop

The algorithm, explained. Start with a pretrained model and a stack of problems whose correct answers you know. There are no worked solutions, only the answers. For each problem, the model writes out its reasoning and ends with an answer. If that answer matches the key, the reasoning is kept. If it misses, the model gets one more try with the correct answer shown as a hint; if it now reasons its way there, the reasoning is kept and the hint is deleted from the record. Finally, the original pretrained model is fine-tuned on everything kept, and the loop starts over with a slightly better model writing the next round of reasoning.

correct ✓ keepwrong ✗✓ kept↺ repeatgeneratea rationale + answerfilteranswer vs. the keyrationalizehint = the answerfine-tunebase model, on keepers

This is STaR's loop.reference 1 It still needs a dataset, questions with correct answers, but nobody has to write the reasoning; the model authors that itself. Everything rests on the answer key: the correctness check that decides which rationales are allowed to train, and the external signal deciding what the model may study.

Algorithm

The same loop from section 1, in ten lines.

M ← M0one model is both student and teacher
loop: until accuracy plateausexpert iteration (Anthony et al. 2017): the same loop, before LLMs
kept ← ∅
for (x, y) in problems:the only supervision that exists: answers, never reasoning
r, ŷ ← M(x)one greedy rollout: low variance, biased exploration; RFT (Yuan et al. 2023) samples k instead
if ŷ = y: keep (x, r, y)in RL terms: a binary reward, checked for free by section 1's answer key
else:
r′, ŷ′ ← M(x, hint = y)hindsight: knowing the destination makes the route easier to find
if ŷ′ = y: keep (x, r′, y)the hint is stripped, so the model trains as if it had found the route unaided
M ← finetune(M0, kept)restart from the base model every round, so early quirks never compound into the weights
The ten lines above are STaR's Algorithm 1, condensed.reference 1 Line 2's loop is not just similar to reinforcement learning: it is the same procedure, expert iteration, from five years before LLM reasoning.reference 2 Line 5 makes one attempt per problem; the RFT variant samples several instead and keeps the ones that reach the right answer.reference 3

2. What the loop boughtZelikman et al. 2022 · CommonsenseQAreference 1

GPT-J, few-shot, answer onlyno reasoning, no training
20.9%
GPT-J, few-shot CoTreasons, learns nothing
36.6%
LaMDA 137B, few-shot CoT23× larger, same recipe
55.6%
GPT-J fine-tuned on human answersthe whole labelled training set
60.0%
STaR without rationalizationits own rationales for 69.7% of the set
68.8%
STaRrationalization reaches 86.7% of the set
72.5%
GPT-3 (30× larger) fine-tunedthe yardstick: 175B vs 6B
73.0%
GPT-J has 6B parameters, and nobody wrote it a single worked solution. Fine-tuned on rationales it generated and filtered itself, it lands at 72.5%, clear past the 55.6% of a LaMDA 23× its size prompted the same way, beside 73.0% for a GPT-3 30× its size fine-tuned on the full human-labelled training set.reference 1 And the gap above the 60% answer-only fine-tune is the page's point. The value was never the answers, it was training on the reasoning.

Trained on reasoning, graded on answers. The rationale enters the weights, but only the answer key decides whether it gets there, so a flawed argument that lands correctly isn't just kept, it's imitated.reference 1 That is not hypothetical: models often follow a wrong reasoning path to the right answer.reference 4 Page 9's outcome-versus-process trade, now deciding what the model becomes.

There are now two ways to buy reasoning: spend the compute when the question arrives, or spend it long before, building the reasoning into the model.

  • Spend at answer time. All of Act 2 spends the compute after the question lands (show the road, cap the tokens, sample and vote, score and re-rank, search the tree) and page 11 priced that shelf against parameters.
  • Spend at training time. This page is the other way: it spends long before the question exists, the first knob that lifts the curve without buying a single parameter.

o1 and R1 keep this loop's shape and drop the disguise, so the filter becomes an explicit RL reward on checkable tasks and the model's own sampling becomes the training set. Next comes the frontier, where one lab turns both kinds of spending at once and bills you for each. Page 13 picks it up from there.