Act 1 · Building Intuition

3. Training-Time Scaling

A missing fact needs new knowledge, and the obvious way to supply it is to train the model again. The bill says no, then the data says no. And even if both said yes, it would do nothing about the bias.

1. Challenges in addressing knowledge gaps

a. The bill compoundsderived from the two Epoch trends

Training price is not a fixed number, so every day of the plan costs more than the day before.

010203040500123456789101112the same GPT-4 model, daily · flat $78M a runthe newest frontier model, daily · e.g. Gemini Ultra at $191M · its price climbs$14.2B$17.9Bmonths of keeping it upcumulative bill ($ billions)

total spent after 6 months

Retrain the newest model each day

its price climbs ×2.4 a year, so the total curves upward

$17.9B

Retrain the same model each day

a flat $78M a run, so the total climbs in a straight line

$14.2B

Chasing the frontier costs $3.6B more so far, and the gap widens every month.

Compute per frontier run grows about 5x a year.reference 1 Cheaper hardware absorbs part of that, so the bill per run grows ×2.4 a year.reference 2 Retrain the frontier model of the day, every day, and year one alone runs to $45.5B. Even a flat rate burns $28.5B a year: one GPT-4-class run costs an estimated $78M of compute, and Gemini Ultra an estimated $191M.reference 3

b. It always lags

A training run takes weeks, so the cutoff moves forward but never reaches now. Retraining resets the clock. It does not stop it.

c. It cannot be aimed

There is no retraining for one missing fact. You buy the whole distribution to obtain a single result.

d. The data runs outcited stock, derived demand

Suppose money were no object. It still fails. A bigger model only pays off if it also reads more, in step: about 20 tokens per parameter (one of the numbers inside the model that training adjusts, and the usual way of saying how big it is), which is the scaling law.reference 4 So the ceiling was never money. It is how much there is worth reading, and no budget buys tokens that do not exist.

every quality token humans have writtenreference 5

≈300T tokens

tokens: the pieces of text a model reads, roughly a word each

So when does the field actually reach the ceiling? Epoch project the largest training set forward from its own trend, about ×2.4 a year. Drag the year.reference 6
1T10T100T1Q2024202520262027202820292030300T stockprojected largest training setprojected yearlargest training set (tokens)

By 2028 the trend points at ≈199T tokens, with a 90% band from 64T to 614T. That is 66% of the stock. On this projection the median crosses the stock in Jun 2028, inside the window below.

the trend projection only · Epoch also run a compute-based projection, which is not drawn here

todayfully utilized 2026–2032 · median 2028

The band is the cited window, not a burn rate; nothing here depletes.

2. Challenges in addressing knowledge bias

Page 2's model held the 2023 title the whole time and dropped it anyway, so the miss was never about what it had read — more reading cannot answer it. And because a model settles on whatever its corpus says most, more text from the same world only sharpens that default rather than loosening it.

Training-time scaling buys more facts, later, at compounding cost, and it cannot touch a bias that was never about missing facts. Every wall here limits what a model can be given before the question arrives, which leaves only what it does after it.