how a 70B model fits in 4 GB of VRAM, and what each token costs
What you will learn
Layer-streaming inference keeps one layer of the model on the GPU at a time and reads the
rest from disk, for every token. VRAM stops being a function of model size: a
70B model fits in 4 GB.
The price is time and disk traffic. Generating one 16-token sentence this way, at 4 GB on an NVMe,
reads about 2.2 TB off the disk and takes
~15 minutes. With every layer resident (a 142 GB card),
the same sentence takes ~2 s: four hundred times faster,
same model.
What to do
Pick a GPU size, a model and a disk. Press Generate and watch the layers travel,
token by token, at the speed the hardware dictates. The meters count real disk traffic.