What is FP4, and how much memory does a model need?
Precision is the lever. A weight is a number, and numbers have a size. Count the parameters, multiply by the bytes each takes at FP16, FP8, FP4 or NVFP4, and you can read a model's memory bill, and see why mixture-of-experts changes what you pay per token.
Serving cost is set less by how many parameters a model has than by how many bytes must move through the chip per token. Precision sets the bytes per weight; mixture-of-experts sets how many weights move at all.
A weight is just a number, and numbers have a size
Every parameter in a model is a single number, and a number costs memory in proportion to its precision, the count of bits used to write it down. At FP16 each weight takes two bytes, at FP8 one byte, at FP4 half a byte. The format is a choice, and it is the main lever on how much memory a model needs.
This is why precision shows up in every frontier chip announcement. Cutting the bytes per weight is the cheapest way to make a model both fit and run faster, because during decode the chip streams those weights again for every single token it writes.
Count the weights, read the memory bill
Take a 550-billion-parameter model. At FP16 its weights alone need about 1,100 GB of memory, that is 550 billion times two bytes. At FP8 that halves to 550 GB. At FP4 it halves again to roughly 275 GB. Same model, same knowledge, a quarter of the memory, purely from writing each number with fewer bits.
For scale, one Blackwell-class GPU holds around 192 GB of high-bandwidth memory. So even at FP4 the weights do not fit on a single chip, and that is before the KV cache for a live conversation. Serving a frontier model is inherently a multi-GPU problem, which is what makes the interconnect matter so much.
Memory for weights is just parameter count multiplied by bytes per weight. Nothing more exotic than that.
Moving FP16 to FP8 to FP4 halves the footprint each step: 1,100 GB, 550 GB, 275 GB.
At 275 GB the weights alone overflow a single GPU, before you add the KV cache. Serving is a multi-GPU job.
Dense vs mixture-of-experts: you do not load the whole brain
The memory bill above assumes a dense model, where every parameter is used for every token. A mixture-of-experts model breaks that assumption. It holds many specialised expert sub-networks, and a small router sends each token to only a few of them, so the model can carry far more total parameters than it pays to run on any single token.
That splits the number that matters into two. Total parameters set how much must live in memory across the cluster; active parameters set how many bytes actually stream through the chip per token, and therefore the speed and cost. A model can hold 670 billion parameters yet activate only 37 billion per token, buying the knowledge of a huge model at the serving cost of a small one. Nvidia's Nemotron pushes this further with a Latent MoE that compresses each token before it crosses the interconnect, fitting roughly four times the experts at the same cost.
Cell counts and parameter figures are illustrative.
MoE holds the knowledge of a huge model but only pays to run a small slice of it on any single token.
A small network sends each token to a few experts. Which ones is impossible to predict in advance, which stresses the interconnect.
Decode bandwidth tracks the active parameters, not the total. That is why MoE serves cheaply despite a giant parameter count.
Source: Bryan Catanzaro (VP, Applied Deep Learning Research, Nvidia), The MAD Podcast with Matt Turck, July 2, 2026.
Precision is a bandwidth lever, not just a capacity lever
Fitting the model is only half the payoff. Because decode is memory-bandwidth-bound, the chip spends most of each token just pulling weights out of memory. Halve the bytes per weight and you halve the bytes that must move, so the same hardware writes tokens faster and cheaper. Precision is a lever on both memory capacity and memory bandwidth at once.
This is the through-line behind a decade of chip generations: Hopper made FP8 practical, Blackwell made FP4 practical, and each step let the same silicon serve a bigger model at a lower cost per token. The precision ladder and the bandwidth wall are the same story told from two ends.
The floor: you cannot shrink the bits forever
There is a limit. Below a certain precision the rounding error in each weight starts to corrupt the model's output, so accuracy falls off a cliff. The frontier is not simply fewer bits, it is fewer bits without losing quality. NVFP4, the format on Blackwell, is 4-bit numbers paired with a per-block scaling trick that preserves the important range, which is what lets a model run at half a byte per weight and still hold its accuracy.
So the real competition is not who can quantise hardest, but who can push precision lowest while keeping the model honest. That is why a numeric format ships as a headline feature: it is one of the few levers that improves capacity, bandwidth, and cost together, right up until it breaks.