# Page 9, "Outcome vs Process Supervision" — the run behind the figures

Everything page 9 draws, plus the code that drew it and every chain it rests on.

```
page-09-outcome-vs-process/
  page_09_outcome_vs_process.ipynb   the notebook: the checker, the gates, every figure
  harness.py                         shared plumbing (model loading, step spans, resume log)
  requirements.txt
  out/page_09/
    traces.jsonl                     all 1,580 graded chains, one JSON object per line
    derived.json                     the aggregates the page plots
    manifest.json                    models, decoding, item set, versions, hashes
```

## Re-draw the figures without a GPU

This copy opens in `RUN_MODE = "REPLAY"`, which loads no model at all.

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
jupyter lab page_09_outcome_vs_process.ipynb
```

Run it from this directory, with `harness.py` beside the notebook. REPLAY re-runs the arithmetic
checker over all 18,772 steps from the trace text, re-fits the PRM's operating point, and
recomputes every rate, gate and figure. It reproduces the numbers on the page exactly.

**One thing it does not recompute: the PRM's raw scores.** Those need 15 GB of weights and a
16 GB card, so REPLAY adopts them from `traces.jsonl` and says so in its output. The adoption is
keyed on the checkpoint, its revision, the quantisation and where the steps were cut; if any of
those differ from the run that made them, the notebook prints both keys and refuses rather than
mixing two measurements. Set `PRM_ADOPT_SCORES = False` to force a rescore on a machine that can.

`manifest.json` records which half is which: `graders.prm.scores` is `"adopted"` or
`"computed by this run"`, and `scores_adopted_from` names the run that computed them. The
separation preflight is carried forward the same way and flagged, because it needs the model.

## How this run was published

Two stages. A `FULL` run on a 16 GB card produced the traces and the scores. A `REPLAY` run then
re-derived every published aggregate from them and wrote `derived.json` and `manifest.json`. That
is why the manifest records `run_mode: REPLAY` while the scores come from a `FULL` run, and it is
what lets `manifest.json` name the notebook in this download rather than a working copy you cannot
obtain. Two independent `FULL` runs produced byte-identical traces before this was published, so
adopting scores is equivalent to recomputing them on that hardware rather than a hope.

## What the run is

Qwen2.5-3B-Instruct chains reused from page 7's pile at temperature 0.3, top-p 0.95, a 2,048-token
cap, on the same 100 MATH-500 level 4 and 5 items pages 5 to 8 use. Only chains the model ended
itself are graded: a chain we cut off and forced to commit is broken by us, and counting that break
as bad reasoning would report our own experiment as a finding. Section 3 applies that filter and
measures what it dropped.

Every chain is graded twice. A deterministic arithmetic checker judges only claims that are
closed-form numeric on both sides, with no symbols, so every label it hands back is a fact you can
redo by hand. That precision costs recall: 94.2% of steps carry no such claim and come back
`unknown`, and 1,062 of the 1,580 chains contain not one step it can check. The second grader is
`Qwen2.5-Math-PRM-7B` at 8-bit, admitted to the page only after clearing a pre-registered gate
against the checker (AUC 0.89), at a cut-off fitted on the checker's own labels. The two are
therefore not independent readings of the same thing, and section 6 says so.

Grading the final answer is `extract_final_candidate` then `grade_answer` from chapter 3 of
Sebastian Raschka's [reasoning-from-scratch](https://github.com/rasbt/reasoning-from-scratch), used
unchanged, so answer grading is not ours to defend. Every number on the page is ours.

Sampling was seeded, and floating-point reduction order shifts with batch size and hardware.
Re-deriving reproduces the shape and the ranking; whether your hardware reproduces every digit is
what REPLAY cannot tell you.
