Act 2 · The Toolbox
Download the notebook (zip, 1.6 MB) ↓9. Outcome vs Process Supervision
Two ways to grade a student: mark the final answer, or mark every line of the working. Where they disagree is where the interesting failures live. Every maths teacher who said “show your work” was defending the second one, and they were right: a correct answer reached by a broken method is a trap, and nobody can tell from the answer alone.
1. What can be graded at all
Three things have to hold before grading a step means anything.
- There has to be a fact. A proof has one at every line; a poem has none mid-stanza. Grade the process where the domain has absolutes, and grade the outcome where it has taste.
- A right answer can be reached by a wrong step. Arithmetic that slips and still lands right, the way a long multiplication can, is exactly what an outcome grader waves through and only a step grader catches.
- The unit is the step, not the word. No word is false on its own, because the claim does not exist until the line is finished, so a grader scores slices big enough to be checkable rather than tokens.
Below, two instruments read these chains with opposite reach. An arithmetic checker recomputes only the 5.8% of 18,772 steps that are closed-form numeric and underlines what it proves; the other 94% state nothing it can compute. A process reward modelreference 1 takes the opposite trade: nothing goes unscored, but a score is a judgment call where the checker's label is deterministic arithmetic.
2. Find the breakreal chains · Qwen2.5-3B-Instruct · graded twice
problem · Find the greatest common divisor of 3339, 2961, and 1491.
An outcome grader reads one thing: the answer. The working is redacted below because that is exactly what this grader sees of it. Switch to Process view to read the working and find the line that breaks.
- 1Step 1 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 2Step 2 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 3Step 3 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 4Step 4 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 5Step 5 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 6Step 6 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 7Step 7 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 8Step 8 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 9Step 9 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 10Step 10 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 11Step 11 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 12Step 12 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
- 13Step 13 is redacted in the outcome view; the outcome grader does not read it. Switch to the process view to read it.not read
final answer → 21 (truth: 21)
👍 Outcome grader: pass. It reads the last line and stops. It cannot see the 1 step below that the checker proves false.
3. What step grading buysmeasured · 1,580 chains
- Outcome grader, 0%.
- It compares the final answer to the known one and never opens the working, so it catches none of this by construction. The question is what reading the steps adds.
- Checker, 0.67%. A floor, not a rate.
- It counts chains, and many chains are the same problem sampled again. By problem it is 5.1%, 4 of the 78 problems that produced a right answer.
- What it could read, 268 of 747.
- It reads arithmetic and nothing else, so 479 of those right chains are silence rather than evidence. Over the ones it did read the rate is 1.87%. Either way it undercounts.
- Qwen2.5-Math-PRM-7B, 19.5%. An opinion, not a proof.
- Its cut is fitted, not held out: the 0.75 that turns a score into a verdict was picked to match the checker's own labels, so these are not two independent readings of the same thing. On Lucky, above, it missed the slip.
4. Where the chain breaksmeasured · 44 localisable chains
What the judge catches, it can place. On the 44 wrong chains with a findable break, the median chain ran 37% of its length before the line that breaks; outcome grading throws all of it away. Read it as a direction, not a digit: on a smaller re-cut sample the four ways of ending a step put that median across 25 points, from 50% to 75%, and none of them landed on the figure quoted here.
And it usually breaks once. Of those 44 chains, 31 contain exactly one false step (70% of them). That is the fact the median cannot carry, and it is the whole case for grading steps: a chain that goes wrong in one place has a place to find, while a chain that was wrong throughout would give a step grader nothing to localise and would cost outcome grading nothing to discard whole. Read the shape and not the digits, and read it at its true reach: 44 of 833 wrong chains (5%) have a findable break at all, so this is the shape of the ones we can place, not of wrong chains in general.