Plurai cuts agent errors 50%

AI agents need a second system that checks whether they followed the rules. On April 29, Plurai launched a tool it says can build those checks from plain-language instructions. (producthunt.com) Those checks are called evals and guardrails: evals score whether an agent did the job, and guardrails block outputs that break policy. Plurai said users describe what an agent should and should not do, then the platform generates training data and deploys a custom small model. (producthunt.com) Plurai said the models are meant to replace “LLM-as-judge” setups, where one large model grades another model’s output. In its launch post, the company said those custom models run in under 100 milliseconds, at 8x lower cost than GPT-as-judge, with more than 43% fewer failures. (producthunt.com) The company tied the launch to a new paper, BARRED, short for Boundary Alignment Refinement through REflection and Debate. The paper was submitted to arXiv on April 28 by Arnon Mazza and Elad Levi. (arxiv.org) BARRED describes a way to train policy guardrails without labeled datasets, which are usually expensive to build by hand. The method starts with a task description and a small set of unlabeled examples, then generates synthetic training data. (arxiv.org) The benchmark in Plurai’s public GitHub repo covers four tasks across three domains: message repetition, privacy disclosure, plan verification, and health advice. The repo lists 158 message-repetition samples, 112 privacy-disclosure samples, 116 plan-verification samples, and 200 health-advice samples. (github.com) Plurai’s launch material said the platform uses a multi-agent debate process to validate the synthetic data before training. That matters because production teams often sample only a fraction of agent outputs when judging with larger models, leaving most interactions unchecked. (producthunt.com) (anthropic.com) Anthropic, in a January engineering post, described a different pattern: run the agent, grade outputs with static analysis where possible, and use large-model judges for behaviors like instruction following. Google Cloud, in March, also argued teams need continuous evaluation instead of ad hoc “vibe testing.” (anthropic.com) (medium.com) Plurai’s pitch is narrower than a full agent-testing stack. It is selling the judge itself: a task-specific checker that can run on every interaction instead of on a sample. (producthunt.com) (arxiv.org) The open question is whether outside users can reproduce the launch numbers on their own workloads. For now, Plurai has put the paper, benchmark repo, demo video, and product launch in public on the same week. (arxiv.org) (github.com) (youtube.com)