Inference as budget battle
An analysis argues inference costs—not training—are becoming the central infrastructure budget fight as organizations move from experiments to sustained use. The piece calls for making model choice, routing and caching visible at the SDK/gateway layer, exposing cost/latency tradeoffs and instrumenting agent loops to detect accidental spend explosions. It frames inference economics as an operational concern that should be governed by the platform rather than left to individual teams. (shashi.co)
Running an artificial intelligence model in production is turning into a bigger budget fight than training one. An April 2026 analysis argued that the expensive part now is repeated inference, the live answering work done on every user request. (shashi.co) Inference is the meter that keeps running after launch: every prompt, every tool call, every retry, and every long context window adds tokens and time. OpenAI’s pricing page on April 13, 2026 listed GPT-5.4 at $2.50 per million input tokens and $15 per million output tokens, while GPT-5.4 nano was $0.20 and $1.25. (openai.com) Anthropic and Google also price the steady-state workload, not just the model itself. Anthropic’s Claude API page listed Claude Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens, and Google’s Gemini Developer API page listed Gemini 3.1 Pro Preview at $2 per million input tokens and $12 per million output tokens for prompts up to 200,000 tokens. (platform.claude.com) (ai.google.dev) That pricing structure changes the internal argument once companies move from pilots to heavy use. A prototype can hide costs in a few demos, but a customer support bot, coding assistant, or agent loop can call models thousands or millions of times a day. (shashi.co) The analysis says that cost control should sit in the software development kit or gateway layer, where a platform team can see and govern model choice, routing, caching, and fallback rules. That is the layer that decides whether a request goes to a premium model, a cheaper model, or a cached result. (shashi.co) Vendors are already exposing those controls in infrastructure products. Vercel’s Artificial Intelligence Gateway documentation says teams can switch providers through one interface, view pricing across providers, and use automatic caching; its caching page says the gateway can add cache markers for providers that require them and use implicit caching for OpenAI, Google, and DeepSeek. (ai-sdk.dev) (vercel.com) Routing is becoming a budget lever too. OpenRouter’s provider routing guide says its default behavior load-balances across providers while prioritizing price, and lets customers sort for throughput instead when speed matters more than cost. (openrouter.ai) Caching matters because many applications resend the same instructions, documents, or conversation history. Anthropic’s prompt caching documentation says caching can cut processing time and reduce costs for repeated prompt prefixes, and its pricing table lists cache hits for Claude Sonnet 4.6 at $0.30 per million tokens, far below the $3 base input rate. (platform.claude.com 1) (platform.claude.com 2) The other hidden bill is the agent loop, where a model keeps calling tools and itself until it finishes a task. LangChain’s LangSmith observability docs say traces record every step of an agent’s execution, including tool calls, model interactions, and decision points, which is the kind of instrumentation the analysis says teams need to catch accidental spend spikes. (docs.langchain.com) (shashi.co) OpenAI, Anthropic, and Google all now advertise lower-cost modes tied to production economics, including cached inputs or batch processing. OpenAI says its Batch Application Programming Interface cuts input and output costs by 50 percent, Anthropic says prompt caching reuses processed prompt sections, and Google says its paid tier includes context caching and a 50 percent batch discount. (openai.com) (platform.claude.com) (ai.google.dev) The argument in April 2026 is not that training stopped mattering. It is that for companies already deploying artificial intelligence, the monthly fight is shifting to the live traffic bill, and the teams that own gateways, budgets, and observability are the ones being asked to contain it. (shashi.co)
