Berkeley study finds 68% short runs

AI agents are supposed to be the part of the stack that finally does things, not just talks about them. But the new Berkeley study is a reality check: once these systems leave demos and hit production, teams keep them on a very short leash. That is the real news here. Not that agents are useless — they are already deployed across finance, healthcare, education, and other domains — but that the winning production pattern looks much more supervised and much less autonomous than the hype suggests. ### What actually came out? The paper is called *Measuring Agents in Production*. Melissa Z. Pan and a large team across UC Berkeley, Stanford, IBM Research, UIUC, and Intesa Sanpaolo surveyed 306 practitioners across 26 domains and added 20 in-depth case studies with agent developers. The paper first appeared on arXiv on December 2, 2025, and the latest listed revision is February 3, 2026. ### Why is the 68% number such a big deal? (arxiv.org) Because it cuts straight through the common picture of an agent as a long-running autonomous worker. In this dataset, 68% of production agents executed at most 10 steps before requiring human intervention. Basically, most teams are not letting agents roam for dozens or hundreds of actions. They are building short runs, bounded workflows, and explicit checkpoints. ### Are teams using fancy custom models? (arxiv.org) Usually, no. The paper says 70% of these production agents rely on prompting off-the-shelf models rather than weight tuning. That tells you something important about where the practical work is happening. Teams are not mostly winning by training bespoke frontier systems. They are winning by wrapping existing models in guardrails, tool access, routing logic, and review steps that make behavior easier to control. ### How are companies judging whether agents work? Mostly with people. The study says 74% depend primarily on human evaluation. That sounds old-fashioned, but it makes sense. Production agent failures are often weird, contextual, and expensive in ways a narrow benchmark misses. If an agent books the wrong thing, sends the wrong email, or takes the wrong action in a regulated workflow, the problem is not just “low score.” The problem is that the system behaved incorrectly in the real world. (arxiv.org) ### So what is the real bottleneck? Reliability. The paper is blunt on that point: consistent correct behavior over time remains the top development challenge. That is a different problem from raw model capability. A model can look smart in a one-shot test and still be a headache in production if it drifts, chains errors across steps, or behaves unpredictably around tools and edge cases. ### Why does human intervention help so much? (arxiv.org) Because production work is not a benchmark — it is a chain of small chances to go wrong. A human checkpoint acts like a circuit breaker. It catches errors before they compound, and it gives teams a way to deploy useful systems before they fully trust them. Turns out the near-term shape of agent adoption looks less like “replace the operator” and more like “speed up the operator with staged handoffs.” That is also consistent with the paper’s broader picture of simple, controllable system design. ### Does this mean the agent boom was overhyped? Not exactly. It means the marketable image of autonomy got ahead of the engineering reality. The study does not say agents are failing everywhere. It says successful teams are making a trade: less freedom, more control. Shorter runs. More prompting. More human review. In other words, the systems that survive contact with production are the ones designed to be dependable, not just impressive in a demo. (arxiv.org) ### Bottom line? The Berkeley result is not “agents can’t work.” It is that real agents already do work — but mostly inside tight boundaries. If you want to understand where this field is actually going, start there: reliability first, autonomy second. (arxiv.org)