Hybrid evaluation finds big blind spots
An engineer reported that relying solely on LLM self‑judges misses a large share of safety violations — their hybrid approach using execution traces, audit logs and snapshots found vanilla LLM judges miss about 44% of safety issues. That implies evaluation should combine behavioral telemetry with model assessments to catch hidden failures. (Lei Li on X)
A lot of artificial intelligence testing still works like grading a student from the final essay alone. A new agent benchmark says that misses what happened during the work, and its baseline language-model judge missed 44% of safety violations even when it could read the full transcript and grader code. (arxiv.org) The basic problem is simple: a language model judge usually sees text, then decides whether the answer looks safe or correct. That works for style and surface mistakes, but agents now click tools, call services, upload files, and leave traces outside the chat window. (arxiv.org) Those hidden traces are what the new system collects. The benchmark records execution traces, which are step-by-step action logs, audit logs, which show what outside services actually received, and environment snapshots, which are saved pictures of the system state after the run. (arxiv.org; paperium.net) Think of execution traces like a delivery driver's route history. A final receipt can say “delivered,” but the route history can show the driver went to the wrong house, and the audit log can show which doorbell camera actually got pinged. (arxiv.org) The benchmark is called Claw-Eval, and it was posted on April 8, 2026. It includes 300 human-verified tasks across 9 categories and scores behavior with 2,159 fine-grained rubric items instead of one broad thumbs-up or thumbs-down. (arxiv.org; paperium.net) The authors call the usual setup “trajectory-opaque” evaluation. “Trajectory” here means the path the agent took, and “opaque” means the judge cannot really inspect that path in a trustworthy way. (arxiv.org) When they compared that usual setup with their hybrid pipeline, the gap was not small. The vanilla judge missed 44% of safety violations and 13% of robustness failures that the hybrid system caught. (arxiv.org) The benchmark also tested 14 frontier models, which means the newest high-capability systems available to the researchers. Performance varied sharply by modality, with most models doing worse on video tasks than on document or image tasks. (arxiv.org) This fits a broader pattern in language-model evaluation research. A 2025 Findings of the Association for Computational Linguistics paper found that fine-tuned judge models can look strong on familiar test sets but still lag on generalizability, fairness, and adaptability. (aclanthology.org) So the takeaway is not that language-model judges are useless. The takeaway is that judging only the answer is like auditing a bank from the printed statement while ignoring the transaction log, the security camera footage, and the vault record. (arxiv.org) If agents are going to book meetings, move money, touch customer data, or trigger real software tools, the evaluation stack has to watch the behavior itself. Claw-Eval’s result is a concrete reminder that a polished final response can hide a dangerous path underneath it. (arxiv.org; cognaptus.com)
