Decagon Launches A/B Testing Suite for AI
Decagon has announced a new experimentation suite for A/B and multivariate testing of AI agents. The platform is designed for rapid iteration on LLM-powered workflows. It provides centralized infrastructure for measuring the impact of changes on both business metrics and user experience outcomes.
- Decagon was founded in 2023 by CEO Jesse Zhang, a Harvard computer science graduate whose previous gaming startup, Lowkey, was acquired by Niantic, and Ashwin Sreenivas, a Stanford masters graduate who previously worked at Palantir. - The company has seen rapid financial growth, achieving a $4.5 billion valuation after its $250 million Series D funding round in January 2026, which was co-led by Coatue and Index Ventures. This valuation tripled in the six months following their Series C round. - A/B testing for LLMs is a complex engineering challenge compared to traditional web testing because of non-deterministic outputs, higher variance requiring larger sample sizes, and the need to balance metrics like accuracy, latency, and token cost. - The new suite enters a competitive MLOps landscape for generative AI, with established players in LLM evaluation and monitoring including Arize AI, LangSmith, and Braintrust. - In the broader AI customer service space, Decagon competes with major incumbents like Salesforce and Zendesk, as well as Bret Taylor's well-funded startup Sierra. - One of the key technical challenges Decagon's platform addresses is "LLM Drift," where underlying models are updated by providers like OpenAI, potentially causing performance regressions that rigorous A/B testing can catch. - The company's primary business is building AI agents for customer support, utilized by companies like Notion, Duolingo, and Substack, with a pricing model based on either a per-conversation or per-resolution fee.
