A page turns into a vector in 20 milliseconds. Eighteen times more text took 32 times longer, and asking the same card to read a 16,000 word history made it request 15 GiB in one block and give up.
Last week I timed one page and then priced it, both on page-length text. The long inputs are the interesting ones: a ninety day user history, a full CTV transcript. So this time I kept the model the same and made the text longer, on the same card. Long text costs more than its share. Take what 1,000 words cost: on the mid-sized model, 42ms inside a page, 40ms at 1,000 words, 44ms at 2,000, 54ms at 4,000, 76ms at 8,000. The 600M model went from 51ms per 1,000 words to 117ms at 32,000, where one text took 3.8 seconds. The same words cost nearly twice as much inside a long text as a short one.
A model does two jobs and their costs grow differently. Reading is the fair one: each word passes through the model’s layers, so twice the words means twice the work. Comparing is the other: every word measures itself against every other word, which is how the model works out what each one means from those around it. Ten words means a hundred comparisons, a thousand words a million, so every time the text doubles, the comparing gets four times bigger. I opened a model up to check. Give it “the river bank was steep” and it builds a seven by seven grid of comparisons, every cell filled, twelve grids side by side at each of twelve layers. The comparisons run one way only: “steep” looking at “bank” scores 0.49, “bank” looking at “steep” scores 0.09.
On a page reading is the bigger cost, so the bill follows the length, as it did last week. Comparing overtakes it once the text passes about 768 words, this model’s working width. The price per word climbed from 4,000 on. Memory suffers for the same reason: the whole grid of comparisons has to sit on the card at once. Sixteen thousand words squared, times sixteen grids, times four bytes a number, is 15.26 GiB in one lump on a card holding 22. Exactly what the crash asked for.
One way around it is to cut the text up and embed each piece on its own. The same 8,446 words cost 645ms in one go, 460ms as two halves, 374ms as four pieces and 336ms as eight. Half the price for the same words, and the result is a vector per session or scene instead of one for everything. It costs something: each piece is embedded blind to the others, so a reference in the last paragraph to something named in the first is lost.
Every model has a ceiling on how much text it will read. Getting near it costs more. The longer the text, the more every single word in it costs.
To turn text into a vector, a model compares every word with every other word
