DeepSeek's V4 slashes KV cache to 2% of normal size, making million-token agents practical

The dirty secret of AI agents is that they break. Not dramatically, but in predictable, boring ways that make long-running tasks a headache. The model runs out of context. The KV cache bloats until the GPU chokes. Tool-call latency degrades halfway through a complex job. DeepSeek’s V4 architecture is a direct, opinionated fix for these specific failure modes, and the numbers are stark.

At a 1-million-token context length, DeepSeek-V4-Pro uses just 27% of the single-token inference FLOPs compared to its predecessor, V3.2. The real headline, however, is memory. The model requires roughly 2% of the KV cache size of a standard grouped-query attention architecture using 8 heads in BF16. It achieves this through a novel hybrid attention system called CSA and HCA, which alternates layers between two compression strategies. Compressed Sparse Attention compresses the KV sequence by 4x and then uses a tiny ‘lightning indexer’ in FP4 to pick relevant blocks, while Heavily Compressed Attention applies a brutal 128x compression and just runs dense attention over the resulting tiny sequence. Both paths store most data in FP8.

This isn’t just a hardware flex. The paper outlines specific post-training choices that make agents work better in the real world. The most practical is ‘interleaved thinking,’ which preserves the model’s reasoning chain across multiple user turns during a tool-use session. In V3.2, a new user message would wipe the model’s internal scratchpad, forcing it to re-derive its plan. V4 keeps the full chain of thought intact until the tool-use conversation ends, enabling coherent state over hundreds of steps.

DeepSeek is open-weighting this under a permissive license and publishing a technical report that feels more like infrastructure documentation than a typical glossy paper. The bet is clear: the next bottleneck for useful agents isn’t a benchmark score, it’s the economics of sustained inference.