AT&T Slashes AI Costs 90% by Rebuilding Agentic Stack

AT&T, now processing 8 billion tokens a day, re-architected its agentic AI stack and cut LLM API spend by 90%. The new system decouples agent orchestration from model serving, using specialized agent roles and focusing on observability to manage costs at enterprise scale for tasks including insurance claims and network ops.

AT&T's move to a decoupled architecture reflects a broader trend in designing multi-agent systems for scalability and maintainability. By separating agent logic from coordination, individual agents can be updated or replaced with minimal disruption to the overall system. This modularity is crucial for complex, high-stakes workflows, allowing for specialized agents to handle distinct tasks like data analysis, report generation, or safety monitoring. This architectural shift is enabled by frameworks that treat agents as specialized, callable tools within a larger, orchestrated system. Frameworks like CrewAI, LangGraph, and AutoGen each offer different approaches to this coordination, focusing on role-based models, graph-based workflows, or conversational collaboration, respectively. The core principle is to move beyond a single, monolithic agent to a distributed system where specialized components collaborate to solve complex problems. In insurtech, this multi-agent approach is being applied to automate and enhance the entire claims lifecycle. AI agents can now handle tasks from First Notice of Loss (FNOL) and data extraction using OCR to fraud detection and even processing payouts for high-volume, low-complexity claims with minimal human intervention. AI-driven underwriting platforms are also leveraging these techniques to analyze vast datasets, leading to more accurate risk assessment and personalized policy pricing. A key component of managing these large-scale AI systems is a focus on observability. Unlike traditional monitoring, AI observability tracks specific metrics like token usage, model drift, and response quality to provide insights into the internal state and behavior of the system. This allows teams to detect and diagnose issues like data drift or bias before they impact business outcomes. For technical leaders on a Principal IC track, this trend highlights the importance of systems thinking and architectural vision. The ability to design and influence the adoption of scalable, modular architectures is a key differentiator for senior individual contributors. This involves not only deep technical expertise but also the ability to communicate trade-offs and guide teams toward long-term, sustainable solutions. The venture landscape in insurtech reflects this technological shift, with a strong focus on AI-centered investments. While overall funding has seen a correction from its 2021 peak, there is a growing interest in B2B SaaS platforms that offer automation and AI-driven analytics. Reinsurers, in particular, are becoming more active in mid-stage funding rounds, indicating confidence in more established insurtech ventures.

Get your own daily briefing

Scout delivers personalized news, insights, and conversations tailored to your role and industry.

Download on the App Store

Shared from Scout - Be the smartest in the room.