TurboQuant slashes memory needs
Google’s TurboQuant algorithm reportedly compresses KV cache 6x to ~3 bits and delivers ~8x faster throughput on Nvidia H100s with no accuracy loss — a major efficiency lever for inference costs. (x.com)
Google Research published a TurboQuant explainer on March 25, 2026 and linked the work to a formal paper listed as arXiv:2504.19874 (original arXiv posting Apr 28, 2025); the paper’s author list includes Amir Zandieh, Majid Daliri, Majid Hadian and Vahab Mirrokni and the work is slated for presentation at ICLR 2026 in Rio de Janeiro (Apr 23–27, 2026). (research.google) The TurboQuant pipeline combines three named stages—PolarQuant, a QJL stage, and an online vector quantizer—and the blog and paper describe PolarQuant as applying a random rotation before coordinate-wise quantization to enable unbiased inner‑product estimates at inference time. (research.google) The paper supplies theoretical distortion‑rate bounds for its online vector quantization approach and reports that TurboQuant outperforms standard product quantization on nearest‑neighbor recall while also dramatically reducing indexing time in their experiments. (arxiv.org) Open‑source community work has already surfaced: multiple PyTorch implementations and ports into llama.cpp and other inference stacks are available on GitHub, and a PyPI package and independent repos report experimental validation on real models such as Qwen2.5‑3B (tests including consumer GPUs like an RTX 3060 appear in community repos). (github.com) Independent coverage and technical commentary highlight that TurboQuant was framed to solve inference‑specific constraints (online compression, per‑request distribution shifts) but note that production serving validation and integration with existing serving stacks remain work‑in‑progress according to analyst write‑ups. (yage.ai)
