LangSmith launches proactive agent engine

LangChain said on May 13 that it launched LangSmith Engine, a new product in public beta that turns production trace data into a workflow for identifying, diagnosing and fixing agent failures. The company said the tool watches traces from LangSmith projects, groups recurring failures into named issues, diagnoses likely root causes and proposes fixes and evaluation coverage for review. (docs.langchain.com) LangSmith is LangChain’s platform for observing, evaluating and deploying AI agents and LLM applications, according to the company’s product pages. (docs.langchain.com) The launch adds an “Engine” layer to an existing stack that already included observability, evaluation, deployment, Insights and Polly, LangChain’s in-product assistant for analyzing traces and runs. The product is aimed at a problem LangChain has described in its own materials: teams often have to read large volumes of traces manually to find patterns, decide what to fix and then create tests so the same failure does not return. (langchain.com) LangChain said Engine automates that loop by using production traces as the starting point for issue detection and follow-up work. ### What does LangSmith Engine do once traces start coming in? LangChain said Engine “watches your production traces, clusters failures into named issues, diagnoses root causes against your code, and drafts PRs and evaluators for your review.” In the docs, the company describes a closed loop in which a recurring failure is detected, a root cause is diagnosed, a fix is proposed, an evaluator is deployed to catch regressions and the issue is reopened automatically if it returns. (langchain.com) The docs say each issue can include the relevant traces, a proposed fix, a custom evaluator for future regressions and ground-truth dataset examples generated from production trace inputs for offline evaluation. (langchain.com) GitHub repository connection is optional, but LangChain says repository access lets Engine use source code to diagnose problems and generate fixes. ### How is this different from LangSmith’s earlier trace-analysis features? LangChain had already shipped tools that analyze traces, including Insights and Polly. (langchain.com) Insights automatically analyzes traces to detect usage patterns, common behaviors and failure modes, and it organizes results into categories and subcategories with executive summaries and percentages showing how often patterns appear, according to the docs. Polly, also already part of LangSmith, lets users ask natural-language questions about traces, threads, prompts and datasets from inside the workspace. (docs.langchain.com) LangChain says Polly can browse runs across a project, analyze individual traces and help users create datasets or inspect failures without paging through logs manually. LangChain presents Engine as the next step beyond those tools. The May 13 announcement says the product not only finds patterns in trace data but also proposes remediations and evaluation coverage, with users reviewing and merging improvements rather than starting from a blank page. (docs.langchain.com) ### What setup does LangChain require? The LangSmith Engine docs say users begin in the Issues tab inside a tracing project. From there, they can optionally connect a GitHub repository, select priority categories such as tool-call failures or latency, choose a model provider and save provider API keys as workspace secrets before clicking “Start analyzing.” (docs.langchain.com) The same docs say Engine uses two models from the selected provider: a heavier model that reasons over traces and writes the agent overview and issues, and a lighter model that screens trace batches before deeper analysis. (langchain.com) Before issues are surfaced, Engine generates an overview document describing the project’s purpose, architecture and key metrics from traces, which users can review and edit. ### What data does LangSmith use as the raw material? LangSmith structures observability data around projects, traces, runs and threads, according to the documentation. (docs.langchain.com) A trace records the sequence of steps for a single operation, runs represent units of work such as model calls or retrieval steps, and threads link traces across multi-turn conversations. LangChain says traces are the core record of what an agent did and why, and it has argued in product materials that agent teams need that trace data to localize failures, measure latency and cost, and build evaluation datasets from real production behavior. (docs.langchain.com) Engine is built on top of that existing tracing setup rather than requiring a separate system, the company said. LangChain said on May 13 that LangSmith Engine is available in public beta, while the product docs describe it as “under active development” and direct users to contact the LangChain team for feedback or access. (docs.langchain.com) The company’s site links the release to its May 13-14 Interrupt conference and to LangSmith tracing projects where the feature is enabled through the Issues tab. (langchain.com 1) (langchain.com 2)