Kubernetes benchmark finds reasoning gaps

Brandon Foley’s May 8 CNCF benchmark tested AI coding agents on real Kubernetes bug reports and found that the tools could often locate the right files and patch an immediate defect. The study said the same agents regularly missed follow-on changes needed elsewhere in the codebase, producing fixes that were locally plausible but incomplete at cluster scope. Foley used open pull requests from the Kubernetes repository as the benchmark and gave agents only the issue descriptions, not the pull request diffs or descriptions. InfoQ reported the results on May 15, saying the findings challenged the idea that better retrieval alone would materially improve automated bug fixing in large distributed systems. ### Which agents did Foley actually compare? Foley’s benchmark compared three configurations against Kubernetes bugs spanning kubelet, scheduler, networking, storage and apps, according to the CNCF post. One setup used retrieval-augmented generation only through KAITO RAG Engine backed by Qdrant. A second used a hybrid approach that started with RAG and then allowed local filesystem access. A third used only a local clone of the repository with direct filesystem search. (cncf.io) Claude Opus 4.6 was the model in all three configurations, and Foley held the timeout and output format constant across runs. The CNCF post said each session had a five-minute limit and that the only variable was how the agent could access code. Foley also required the RAG and hybrid agents to query retrieval before attempting a fix so the comparison would measure retrieval strategy rather than let the hybrid agent bypass it. (cncf.io) ### What broke when the code looked correct? Foley wrote that the initial assumption was that success would largely depend on retrieval: if an agent found the right code, it should be able to generate the right fix. The benchmark did not support that assumption. The CNCF post said agents often surfaced the right files but still failed to connect changes across them, misread the true scope of the issue, or produced fixes that were “locally plausible but globally incorrect.” (cncf.io) InfoQ said the dominant failure mode was incompleteness rather than an obviously wrong patch. In its account of Foley’s results, agents fixed the main bug while overlooking adjacent changes, omitted updates in dependent integration logic, or stopped after finding a partial fix already present in the codebase. InfoQ summarized the pattern this way: the agents did not reliably ask what else had to change once the immediate issue appeared resolved. (cncf.io) ### Did retrieval help at all? InfoQ reported that retrieval changed speed and code-discovery behavior. In the benchmark, the RAG-only setup was the fastest at an average of 76 seconds because it skipped filesystem navigation, while the hybrid setup averaged about two and a half minutes and was the slowest because the required RAG-first phase added steps before local exploration. The CNCF post and InfoQ both said those differences did not translate into a clear reasoning advantage on multi-file or system-level fixes. (infoq.com) Foley wrote that the bottleneck was “not just finding code” but reasoning over code in context. That distinction is central to the benchmark: retrieval could help agents discover relevant files, but it did not reliably make them trace dependencies across the broader Kubernetes system. ### What did the benchmark say about architecture choices? InfoQ reported a second pattern in Foley’s tests around design decisions. When agents had a choice, they tended to add new abstractions instead of reusing existing ones. In one case, InfoQ said, the correct fix used an existing `RestartCount` field, while agents introduced a new `Attempt` field that worked functionally but added architectural weight. (cncf.io) That detail matters because Foley’s benchmark used real Kubernetes pull requests rather than toy examples. The CNCF post said the cases ranged from a one-line guard clause to a roughly 900-line multi-file refactor, which meant the agents were being tested inside a large, active codebase where correctness depended on how a change fit with surrounding components. (infoq.com) ### Where can readers check the benchmark themselves? The CNCF blog post published on May 8 lays out the benchmark design, the three retrieval configurations and the Kubernetes pull-request-based test cases. InfoQ’s May 15 article summarizes the same study and names Brandon Foley as the author of the original benchmark. Those two publications are the current public record for the experiment and its reported results. (cncf.io)