Models · 9 of 10

The Context Wall (FlashAttention, GQA, & SSMs)

Long context feels like memory. Under the hood it is a live working set that has to be read, routed, cached, and paid for.

Where the binding constraint sits today

Context becomes binding when the application wants more state than the model can read cheaply at low latency. The constraint is memory movement, not the marketing size of the window.

Context is not free recall

Putting more text into a prompt does not give the model magical memory. It gives the model more material to process at inference time.

The system has to tokenize it, route it through attention, store or recompute cache state, and pay the latency and memory cost.

The KV cache is the hidden bill

During generation, the model keeps key and value states so it can attend back to prior tokens without recomputing everything. That cache grows with context length, layers, heads, and batch size.

At long context, the cache can dominate memory pressure. That is why serving long documents is a systems problem, not just a model feature.

The cache lives on a memory ladder

Every storage tier in a serving stack has a property called drain time: how long it would take to read the whole tier, end to end, at full bandwidth. HBM drains in about 20 milliseconds. DDR drains in seconds. Flash drains in roughly a minute. A spinning disk drains in roughly an hour. Each tier earns its place when its drain time roughly matches how long the data needs to stick around.

The five-minute tier is almost certainly flash. The one-hour tier is almost certainly spinning disk. The point for product design: long context is not free, but it is also not all priced the same.

Memory ladder, by drain time

Tier

Drain time

Useful retention

HBMfastest, on the chip

~20 ms

one step

DDRsystem memory next to the chip

~1–10 sec

seconds

Flashwhere the 5-minute cache likely sits

~1 min

minutes

Spinning diskwhere the 1-hour cache likely sits

~1 hour

an hour

Each tier has a drain time = capacity ÷ bandwidth. A tier earns its keep when the retention window roughly matches its drain time. Cache pricing on LLM APIs reads cleanly through this ladder.

FlashAttention & Grouped-Query Attention (GQA)

To bypass the memory wall, modern models use architectural optimizations. FlashAttention avoids writing intermediate attention matrices to slow GPU global memory by keeping them in fast GPU SRAM.

Grouped-Query Attention (GQA) groups key and value heads, allowing multiple query heads to share a single key/value pair. This drastically shrinks the KV cache size (often by 8x) with negligible loss in accuracy, extending the limits of prompt caching.

State-Space hybrids (SSM / Mamba)

Standard attention is quadratic: doubling the sequence length quadruples the attention compute and memory overhead. State-space models (like Mamba) compress the context into a fixed-size running summary. This scales linearly, bypassing the KV cache overhead entirely, though it can lose the ability to pinpoint exact individual tokens over extreme lengths.