Act 2 · The Toolbox
the search arm, written and not run ↓10. Searching Through Thoughts
A model writes one word at a time. At every word it has a few candidates and a hunch about how likely each is, and it picks one. Always pick the likeliest and you have a chain: a single line, written forward, with no way back. Keep a few unfinished lines alive instead and you have a tree.
Nothing here is a measurement of ours. Pages 5 to 9 each ship the run their numbers come from. This page ships none. The 29-word tree below is written by hand, chosen so the five behaviours are legible, and the crossword bars in step 3 are Yao et al.'s.
1. Finish the line
Be the model for four words. Below is a couplet missing the end of its second line, with the words the model thinks could come next and its hunch about each. Tap a word to add it. The finished line has to rhyme with “night”. That is the whole rule.
I couldn't sleep at all last night,
so I ___ ___ ___ ___
what the model thinks comes next
0 of 4 words
The trap is not a toy. Run for real on GPT-2, always-take-the-likeliest-word gets stuck repeating a phrase in 73.66% of its continuations.reference 1 Human text: 0.28%.
2. Watch it search the same tree
Greedy is what you just did: a chain of thought, no checking, no way back. The other four are best-of-N with the judge moved inside the loop, and here the judge is the model’s own confidence: it rates unfinished lines while they are being written, and the search decides which to extend. Nothing on this page checks whether a line can still rhyme. Pick one and press play; the shape of the lit-up region is the strategy.
Take the likeliest next word every time and never look back. This is plain autocomplete, and it is what a chain of thought does: one line, written forward, no way to undo a word.
0 / 29
words the model rated
the compute bill
Press play to watch it search.
The words and their probabilities are written by hand. The search over them is real, and the replay is its actual trace.
3. What search buys
Running these five head-to-head on a real model is out of reach here. Once a step is a sentence rather than a word, breadth-first has to keep every line a model could write next, depth-first cannot tell a line is doomed without finishing it, and MCTS needs thousands of finished lines per question; only greedy and beam stay practical. That is a limit of the hardware, not a verdict on the algorithms. The closest measurement takes the same branch, score, backtrack machinery apart piece by piece on a real word task: GPT-4 filling 5×5 crosswords, with and without the ability to back up.reference 2 The two bars with a piece switched off are the ones that matter.
Chain of thought fills 15.6% of the words, barely above no reasoning at all, because one chain commits to its first answer just as yours did. Take the full search and switch off pruning and it drops to 41.5%. Switch off backtracking instead and it collapses to 20%, almost back to plain chain of thought. That is the whole argument in two bars: what search buys is not more thinking, it is the ability to take a thought back. What that costs is the trade the rest of this act is about.