AI SRE evals: the hard problem
An SRE practitioner warned AI SRE agents still struggle with evaluation because they must fetch real‑time data across multiple systems and infer contextual root causes — he called it the hardest problem in the space. (x.com). That critique flags a major gap for teams trying to push AI into production incident response and reliability tooling. (x.com)
Several vendors advertise sub‑minute root‑cause identification from agentic SRE products, with Traversal saying agents surfaced root causes in under a minute on real customer incidents. (traversal.com) Microsoft engineers who built an Azure SRE Agent reported starting with “100+ tools and 50+ specialized agents” and deliberately consolidated to five core tools and a few generalist agents to make in‑production agents more reliable. (techcommunity.microsoft.com) Microsoft Research notes automated root‑cause analysis remains “demanding” because RCA requires deep, service‑specific domain knowledge and access to heterogeneous runtime telemetry. (microsoft.com) Observability vendors are layering generative‑AI context features into RCA workflows—BigPanda says it now derives situational context and produces a causal confidence score for correlated changes, and Grafana released unified contextual RCA workflows at ObservabilityCON 2024. (bigpanda.io) Smaller incident‑ops vendors make large claims: Rootly advertises up to a 90% cut in response and resolution time using AI SRE features, while NeuBird markets “real‑time diagnosis and remediation” for hybrid and multi‑cloud stacks. (rootly.com) Independent analysis finds a stark gap between pilots and production: RAND Corp. reported that more than 80% of AI projects fail to reach production, underscoring why practitioners call evaluation and real‑world validation a core obstacle. (rand.org) Academic work and industry engineering both point to the same next steps—in‑context learning for RCA and tighter tool consolidation to guarantee reliable, up‑to‑date cross‑system data access in agents. (arxiv.org)
