Agents Struggle to Optimize GPU Code

The push for agent-driven code optimization stems from a critical problem in production AI: severe underutilization of expensive GPU hardware. Many large language model workloads only achieve 30-50% of a GPU's theoretical peak efficiency, often bottlenecked by memory bandwidth saturation during the attention step, rather than raw compute limits. To measure this, ISO-Bench uses 54 real-world optimization tasks sourced from the open-source projects vLLM and SGLang. The benchmark evaluates agent-generated code patches on powerful NVIDIA H100 GPUs, focusing on concrete performance gains in demanding inference scenarios. The choice of vLLM is significant; it's a high-performance serving engine from UC Berkeley's Sky Computing Lab designed specifically to combat memory bottlenecks. Its core innovation, PagedAttention, mimics virtual memory to manage the memory-intensive key-value cache, making it a prime example of expert-level GPU optimization. ISO-Bench's evaluation is twofold, using both "hard" and "soft" metrics. Hard metrics include direct measurements of speed improvements like Time to First Token (TTFT) and throughput, while soft metrics use an LLM-as-a-Judge approach to assess if the agent correctly identified the performance bottleneck in the first place. This research highlights a crucial gap in the capabilities of current AI coding agents. While tools like GitHub Copilot and Claude are evolving from simple autocompleters to more autonomous systems, they still fall short on complex optimization tasks. Prior benchmarks like KernelBench have also shown that even top models succeed on fewer than 20% of GPU kernel generation tasks. The core challenge revealed by ISO-Bench is that successfully identifying a performance issue does not guarantee an effective solution. Agents often propose fixes that are functionally correct but fail to deliver real-world speedups, a critical distinction for ML engineers managing the cost and latency of production inference pipelines.