Infra limits: power and interconnects
- An industry expert noted Dell currently caps clusters at roughly 10,000 GPUs due to practical limits. (x.com) - Grid capacity figures like 300–400MW limits were cited as material constraints for scaling to multi‑GW AI sites. (x.com) - Engineers also stressed that GPU‑to‑GPU interconnects (NVLink/InfiniBand) matter more than raw GPU speed for distributed workloads. (x.com)
Training a giant AI model is now as much a networking-and-power problem as a chip problem. Dell has said practical limits still cap many clusters at about 10,000 graphics processors, or GPUs. (dell.com) That 10,000-GPU figure is not theoretical. Dell’s November 29, 2023 deal with Imbue called for a PowerEdge cluster with “nearly 10,000” Nvidia H100 GPUs, a scale that has become a reference point for what one vendor can assemble as a single system. (dell.com) The bottleneck starts with electricity. The U.S. Energy Information Administration said on January 13, 2026 that U.S. power demand is in its strongest four-year growth period since 2000, with data centers a major driver. (eia.gov) Large AI facilities also run into site-level limits before they run out of demand for chips. The International Energy Agency said in April 2026 that there is “no AI without energy,” specifically electricity for data centers, and its latest analysis ties AI growth directly to rising power demand. (iea.org) The second constraint is how fast GPUs can talk to each other. In distributed training, chips constantly exchange model updates through collective operations such as all-reduce, which Nvidia’s NCCL documentation describes as a core pattern for multi-GPU workloads. (docs.nvidia.com) That is why interconnects such as NVLink and InfiniBand matter so much. Nvidia’s GB200 NVL72 system links 72 Blackwell GPUs inside one NVLink domain and advertises 130 terabytes per second of low-latency GPU communication in a single rack. (nvidia.com) Nvidia’s own system documentation shows what that means in hardware terms. An NVL72 rack combines 72 GPUs, 9 NVLink switch trays, power shelves, bus bars, and liquid-cooling manifolds in one rack-scale design. (docs.nvidia.com) Once training spills beyond one tightly linked rack, the cluster leans on a fabric such as InfiniBand to connect servers. Nvidia’s networking material and NCCL guidance both frame that fabric as essential because communication time can dominate when models are split across many nodes. (developer.nvidia.com) Cooling is part of the same equation. Dell said in August 2024 that new AI Factory server designs were adding higher-power support and liquid-cooling options as customers pushed toward denser accelerated-computing deployments. (dell.com) The industry’s next wave of systems is getting denser, not lighter. Dell said in May 2025 that its PowerEdge XE9712 with Nvidia GB300 NVL72 is built for rack-scale training, while Nvidia’s current rack documentation shows every NVL72 rack is already a liquid-cooled, power-heavy unit. (dell.com) So the race to build bigger AI clusters is no longer just about buying the fastest GPU. It is about securing megawatts, fitting liquid-cooled racks into real buildings, and keeping thousands of GPUs connected tightly enough that they do not spend their time waiting on each other. (eia.gov)
