Quantization cuts model costs

Quantization shrinks artificial intelligence models by storing numbers with fewer bits, cutting memory use and often lowering inference cost at the same time. (docs.nvidia.com) In practice, that means replacing 16-bit or 32-bit numbers with 8-bit, 4-bit, or even newer low-precision formats, then rescaling them back during computation. ONNX Runtime describes this as mapping floating-point values into an 8-bit space with a scale and zero point. (onnxruntime.ai) The tradeoff is accuracy: squeeze numbers too aggressively and model quality can fall. NVIDIA’s TensorRT documentation says its inference stack now supports signed INT8, FP8, signed INT4, and FP4 formats for quantized values, each aimed at reducing memory traffic and speeding execution. (docs.nvidia.com) That matters because large language models are often limited less by raw math than by how fast hardware can move model weights in and out of memory. NVIDIA says quantization helps by reducing model size and accelerating computation, which is why it is central to current inference tooling. (docs.nvidia.com) The recent wave of interest comes from methods that preserve more quality at lower precision. LLM.int8, published in 2022, described an 8-bit approach for transformer models that keeps a small set of unusually large values in higher precision instead of forcing everything into the same smaller box. (arxiv.org) By 2023, QLoRA pushed the idea further for training and fine-tuning: the paper said a 65 billion parameter model could be fine-tuned on a single 48 gigabyte graphics processing unit while preserving full 16-bit task performance. That result helped turn 4-bit quantization from a niche compression trick into a standard part of open model workflows. (arxiv.org) Open-source tooling followed. Hugging Face’s Transformers documentation says bitsandbytes supports both 8-bit and 4-bit quantization, with 4-bit commonly used alongside QLoRA, making reduced-precision loading available through mainstream model libraries. (huggingface.co) Deployment stacks also widened beyond one vendor. ONNX Runtime positions itself as a cross-platform accelerator for models from PyTorch, TensorFlow, Keras, TensorFlow Lite, and scikit-learn, and its execution providers include hardware-specific backends such as Qualcomm’s QNN and AMD’s Vitis AI alongside general-purpose options. (onnxruntime.ai, onnxruntime.ai, onnxruntime.ai) Hardware support is also moving down the precision ladder. NVIDIA’s TensorRT for RTX documentation says FP8 is supported for matrix multiplications on Ada-generation and newer graphics processors, while FP4 support starts on Blackwell-generation hardware. (docs.nvidia.com) The result is not that every model can be crushed to 4 bits with no downside. The result is that enterprises now have a larger menu of quantized formats, runtimes, and chips, so more models can fit on local servers, workstations, and edge devices than was practical when 16-bit inference was the default. (docs.nvidia.com, huggingface.co)