Inference stack techniques surfaced

Large language model serving has turned into a systems problem: the newest public stack notes focus on routing requests to the right cache, shrinking memory, and splitting work across GPUs. (x.com) A model answers in two stages. It first digests the prompt and builds a key-value cache — a running memory of prior tokens — and then generates new tokens one by one from that cache. (docs.nvidia.com) (bentoml.com) Those two stages stress hardware differently. NVIDIA’s Dynamo docs say prefill is the prompt-processing step, while decode is the token-generation step, and the system can send them to separate worker pools instead of making one GPU do both jobs. (docs.nvidia.com 1) (docs.nvidia.com 2) That split has become one of the clearest knobs in production inference. NVIDIA says its AIConfigurator can compare aggregated and disaggregated deployments and has shown as much as 1.7x better throughput in tested Dynamo setups. (docs.nvidia.com) The cache itself is the other bottleneck. vLLM describes prefix caching as reusing stored key-value blocks when a new request shares the same prompt prefix, which lets the system skip recomputing the shared part. (docs.vllm.ai 1) (docs.vllm.ai 2) For image-and-text models, the same idea extends beyond plain text. vLLM’s implementation notes say multimodal models can use different hashing methods for different input types so cached blocks can be reused across requests with shared multimodal inputs. (docs.vllm.ai 1) (docs.vllm.ai 2) Routing then decides where a request should go. NVIDIA’s KV-aware router picks workers based on cache overlap and decode load, instead of using round-robin or random balancing, so a prompt is more likely to land where useful memory is already warm. (docs.dynamo.nvidia.com) The hardware format in the post points to another lever: storing numbers with fewer bits. NVIDIA introduced NVFP4, a 4-bit floating-point format for Blackwell GPUs, in June 2025 and said it was built to keep more of the accuracy of higher-precision formats while cutting memory and boosting efficiency. (developer.nvidia.com) (developer.nvidia.com) NVIDIA later applied that format directly to the cache. In a December 8, 2025 post, the company said NVFP4 key-value cache quantization cut cache memory use by 50% versus FP8, doubled context length and batch size, and kept benchmark accuracy loss under 1% on the tests it reported. (developer.nvidia.com) Put together, the techniques in the public post describe a familiar 2025-2026 serving playbook: keep shared context resident, send work to the GPU that already has it, compress the cache, and separate prompt ingestion from token generation. (x.com) (docs.dynamo.nvidia.com) (docs.nvidia.com) (developer.nvidia.com)