NVIDIA Blackwell System Shows 100x Inference Gains
NVIDIA's latest GB300 NVL72 'Blackwell Ultra' system is reportedly delivering up to 100x improvements in FP8/FP4 inference benchmarks compared to H100 baselines. Social media discussions suggest this performance leap could drastically reduce costs for agentic AI and long-context tasks on edge and cloud devices.
- The GB200 "Superchip" forms the core of the system, connecting two NVIDIA B200 Blackwell GPUs to one NVIDIA Grace CPU with a 900 GB/s NVLink-C2C interconnect. This creates a unified memory domain, crucial for massive AI and high-performance computing workloads. - Each Blackwell GPU is a dual-die chip, featuring 208 billion transistors manufactured on a custom TSMC 4NP process. The two dies are connected by a 10 terabytes per second (TB/s) link, allowing them to function as a single, unified GPU. - The performance gains are driven by fifth-generation Tensor Cores which add native support for new, lower-precision 4-bit and 6-bit floating-point formats (FP4 and FP6). This allows for significantly higher throughput and reduced memory footprint for inference tasks with minimal accuracy loss. - The full GB200 NVL72 system integrates 36 Grace CPUs and 72 Blackwell GPUs into a single liquid-cooled rack. It utilizes a massive NVLink domain that allows all 72 GPUs to operate as a single, powerful entity. - The B200 GPU itself features 192GB of HBM3e memory, delivering up to 8 TB/s of memory bandwidth. This is a significant increase over the 80GB of HBM3 and 3.35 TB/s of bandwidth found in the previous generation H100 GPU. - To connect the GPUs, the Blackwell architecture uses the fifth generation of NVLink, which provides 1.8 TB/s of bidirectional bandwidth per GPU. This high-speed interconnect is essential for scaling performance across the 72 GPUs in the NVL72 system. - NVIDIA claims the liquid-cooled GB200 NVL72 system can deliver up to 30x faster real-time inference for large language models compared to the H100. For equivalent performance, a single rack of GB200s is claimed to replace 100 racks of H100 systems, with significant TCO and energy savings. - While the GB200 chip consumes more power than its predecessor (1200W vs 700W for the H100), the overall system is designed for greater energy efficiency, delivering up to 25 times more performance at the same power level compared to H100 air-cooled systems.
