Act 2 · The Toolbox
Download the notebook (zip, 3.1 MB) ↓7. Frequency Voting
Page 6 made one chain longer, and ran out of road. So stop asking once: ask the same question many times, and keep the answer that comes up most often.
1. Put it to a votethe idea, before the measurement
You are new in town and want the best pizza. You do not know whose taste to trust, so you do not pick a person: you hand out slips of paper, ask everyone to write down their favourite place, and drop them in a box. Then you count. One enthusiast's odd favourite gets one slip and stays there. The place that is actually good keeps turning up, on slips written by people who have nothing in common.
Nobody had to be right on their own. The crowd only had to be wrong in uncorrelated ways, so the mistakes scatter and the good answer stacks. Done to a model, by sampling the same question many times and keeping the answer that comes up most, that is majority voting, or self-consistency.reference 1
2. One box of slipsone problem · 16 chains · 8 distinct answers
Here is one box. The problem below went to Qwen2.5-3B-Instruct 16 times with sampling turned on, which is what temperature does. Temperature is a dial, written T on every chart on this page: at zero every chain comes back identical, and turning it up makes them disagree more. Every answer it came back with is tallied. Click a row to read the chains that landed there.
What is the average of the two smallest positive integer solutions to the congruence 14u ≡ 46 (mod 100) ?
MATH-500, level 5 · the answer is 64 · picked from 25 problems that qualified
The winning answer holds 9.9 of 16 votes on average across all 100 problems, against 5.1 distinct answers. Agreement concentrates; the rest is a scatter of near-misses, 41% of which are said exactly once.
3. What the vote buysthe same 100 MATH-500 problems, levels 4 and 5 · Qwen2.5-3B-Instruct · one pile, subsampled
That was one problem. Now all 100 of them, with the box holding 1, 2, 4, 8 and finally 16 slips. The chart below draws three lines, and each one answers a different question.
The vote returns the answer is what counting actually gets you: the answer the most chains agreed on. At least one chain is right is the easier question. Did the right answer turn up on any slip in the box, whether or not counting picked it? The gap between those two lines is the right answer being written down and then thrown away. One greedy chain is the baseline to beat: at T = 0, with the temperature dial turned all the way down, sampling is off, which is pure chain-of-thought, also known as greedy decoding. That is what pages 5 and 6 ran, and it makes all 16 chains the same chain.
On 16 chains, the vote returns the answer on 55% of problems, 9% above one sampled chain. The right answer was somewhere in the box on 78%, so 23% of problems had it written down and thrown away. 10% of these votes tie. The bill is 9,931 tokens a problem.
The band widens as you add chains.
The flat line is the point. At temperature 0 every chain is the same chain, so the vote is one voter answering 16 times and nothing is gained. Diversity is the whole mechanism.
The dip at N = 2 is the tie rule. Two chains that disagree are a tie, and a tie returns no answer, so it grades wrong. 52% of two-chain votes tie; by sixteen it is 10%.
distinct answers among 8 chains, by temperature
How many different answers those 8 chains came back with, averaged per problem.
Measured on 30 items. This is a count of answers, not an accuracy. Disagreement is necessary and not sufficient: the vote comes out the same at T = 0.3 and T = 0.6, and past that each chain gets worse faster than the vote recovers it.
4. And what it costsone budget of 2,048 tokens, split N ways
Sixteen chains cost sixteen times the tokens. So what if you do not have sixteen times the budget? Below, one fixed budget of 2,048 tokens is divided between the chains, and the same chains are run again with no budget at all. Read it as two lines, not two experiments: at every point on the x axis the model gets the same number of chains, and the only thing that changes between the lines is whether it is allowed to finish them. They start as one run and end a long way apart.
The gauge under each count is the share of that run's chains that hit the cap and were made to commit mid-thought, which is a property of the run rather than a score.
Dividing a budget does two things at once. One, it adds chains, and two, it shortens every one of them. The shortening is the one that lands. At 16 chains the budget leaves each with only 128 tokens. Every chain hits that ceiling before finishing, and the vote is counting fragments of arguments rather than arguments. Splitting is not what fails, capping is. Let the same chains run to their own end and the vote indeed works, at 9,931 tokens for 16 chains vs 618 for one. There are two ways of spending here, divide one budget, or let every chain finish. The second one is the winner.
5. What the vote threw away
The pile knows more than the count. The right answer was somewhere in the 16 chains on 78% of these problems. The vote returned it on 55%. For the 23% in between, the model produced the right answer and the vote discarded it. This is because counting can only ever return the answer that comes up most, and on 10% of problems it could not even break a tie!