Designing AI agents for production

Manning Publications posted a YouTube talk by Jia Huang on September 1 that framed AI agents as a software engineering problem, not a prompt-writing exercise. The talk, “Designing AI Agents: From Demos to Production,” described a shift toward agents that are reliable, maintainable and ready for production, according to the video description. A second recent conference talk by Meta staff engineer Nishant Gupta made a similar case, saying production systems must provide deterministic guarantees even when large language models are probabilistic. ### Why are engineers talking less about “smart agents” and more about production systems? Jia Huang’s talk was presented as an engineering-focused introduction to agent development, with the stated goal of building systems that are “reliable, maintainable, and ready for production.” The emphasis is on the surrounding software stack — orchestration, state handling, testing and recovery — rather than on model capability alone. (youtube.com) Nishant Gupta’s conference session described the same constraint in more explicit terms: model behavior is non-deterministic, but business workflows still need deterministic guarantees. The session summary highlighted orchestration, workflow state management, retries, checkpoints and failure recovery as core patterns for making agent systems predictable enough to operate in production. (youtube.com) ### What does “deterministic infrastructure” actually mean in practice? Google Cloud’s agent observability documentation says agent systems can “drift, hallucinate, and regress silently,” and says developers need logs, metrics and traces to debug failures, monitor costs and analyze behavior. The same documentation defines traces as records of execution paths and says metrics can be used to monitor latency and token usage. (testmuai.com) That framing matches the production pattern emerging across recent agent talks: keep the model probabilistic, but make the wrapper deterministic. In practice, that means explicit routing rules, bounded tool permissions, versioned prompts, retry policies, checkpointed state and observable execution paths, according to Gupta’s session summary and Google Cloud’s observability guidance. (docs.cloud.google.com) ### Why have evals become a first-class part of agent design? OpenAI’s developer documentation says evaluations “test model outputs to ensure they meet style and content criteria” and calls writing evals “an essential component” of building reliable LLM applications. A separate OpenAI guide for agent workflows says developers should use traces, graders, datasets and evaluation runs to assess agent behavior. (testmuai.com) That pushes agent development away from anecdotal demos and toward repeatable measurement. Under that model, teams test tool selection, task completion, latency, cost and failure modes before changing prompts, models or workflows, rather than relying on a few successful examples. That is an inference from the OpenAI documentation and the production-oriented talks, which both describe reliability as something that must be measured and enforced. (developers.openai.com) ### How are teams handling memory without making agents uncontrollable? LangGraph’s documentation says persistence matters when an agent needs to resume after interruption, recover from failure or remember information across interactions. It says checkpointers persist graph state as checkpoints for short-term, thread-scoped memory, while stores persist application-defined data outside graph state. (developers.openai.com) Anthropic’s engineering note on context engineering draws a related distinction around what information should be curated and maintained during inference. In a separate post, Anthropic said prompt caching can reduce latency by more than 2x and cut costs by up to 90% in some retrieval-heavy setups. ### Where do cost controls and fallback design enter the picture? Anthropic’s current model pages say prompt caching can deliver up to 90% cost savings and batch processing up to 50% savings, giving teams a direct financial reason to structure agent workloads carefully. (docs.langchain.com) Pricing on Anthropic’s Claude pages ranges from $1 per million input tokens for Haiku 4.5 to $5 per million input tokens for Opus 5, with higher output-token prices. (anthropic.com) Those economics reinforce the engineering trend described in the talks: use stronger models selectively, preserve state when possible, and build fallback paths when a task does not require the most expensive model. OpenAI’s API platform and Agents SDK materials also present agents as workflows built around tools and context, not as single-shot model calls. ### What should readers watch next? Google Cloud published new agent observability documentation this week, and OpenAI’s agent-evaluation guides are live in its developer docs as vendors add more production tooling around traces, graders and workflow monitoring. (anthropic.com) LangGraph’s persistence and checkpointing docs remain a reference point for teams building resumable agent flows. (docs.cloud.google.com) (openai.com)