Nvidia, memory and workarounds

Nvidia’s AI story is no longer just about selling more graphics processors; it is increasingly about who captures the profit around memory, networking and custom chips. (nvidia.com) (fool.com) A graphics processor, or GPU, works like a huge team of calculators running the same math at once, which is why it became the standard engine for training and serving large AI models. Blackwell is Nvidia’s current generation, and its memory system is built around high-bandwidth memory, or HBM, a fast stack of memory chips placed next to the processor. (freecodecamp.org) (nvidia.com) That memory matters because AI models must fit their weights, temporary data and user requests somewhere while they run. FreeCodeCamp’s breakdown of Blackwell says memory capacity, bandwidth and latency are separate limits, and Nvidia’s B200 generation is paired with 192 gigabytes of HBM3e per GPU. (freecodecamp.org) (nvidia.com) When one GPU cannot hold a model or enough active requests, cloud operators spread the work across several chips. Nvidia’s answer is to tie many Blackwell GPUs together with NVLink, including an eight-GPU DGX B200 system and the 72-GPU GB200 NVL72 rack, which Nvidia says can act like one large pool for trillion-parameter inference and training. (nvidia.com 1) (nvidia.com 2) That changes the market debate from raw chip demand to the cost of the full system around the chip. The Motley Fool article published April 21 said investors have started focusing less on whether Nvidia can sell accelerators and more on where spending lands as AI deployments move from building models to running them at scale. (fool.com) (nvidia.com) Running models at scale is inference: the stage when a trained model answers prompts, ranks search results or generates code for millions of users. Nvidia now markets Blackwell and Blackwell Ultra around “AI reasoning” and real-time services, while its DGX B200 page says the system delivers 15 times the inference performance of the prior generation. (nvidia.com 1) (nvidia.com 2) The biggest cloud companies are also trying to keep more of that spending inside their own walls. Amazon Web Services says Trainium is designed for training and inference, Google says its sixth-generation Trillium TPU is built to train and serve foundation models, and Microsoft says Maia 100 was designed for large AI workloads in Azure. (aws.amazon.com) (cloud.google.com) (azure.microsoft.com) Those in-house chips are aimed at the same pressure point: lowering the cost of serving AI after the training run is over. Amazon says Trn2 instances offer 30% to 40% better price performance than its GPU-based P5e and P5en instances, and Google says Trillium is more than 67% more energy-efficient than TPU v5e. (aws.amazon.com) (cloud.google.com) Nvidia still controls the broadest commercial AI stack, from GPUs to interconnects to software, and its GB200 NVL72 pitch leans on that integration with 72 Blackwell GPUs and 130 terabytes per second of NVLink bandwidth inside one rack. But the more customers optimize for inference cost, memory fit and internal workloads, the more value can shift into networking, memory packaging and custom silicon built by the clouds themselves. (nvidia.com) (aws.amazon.com) (cloud.google.com) (news.microsoft.com) The next test is not whether Nvidia can sell Blackwell systems; it is whether customers keep paying Nvidia for the whole AI factory as model serving becomes the larger, steadier workload. (fool.com) (nvidia.com)