Inference is the new budget fight
Analysts argue that as pilots end and production use ramps, inference costs — the recurring expense of running models in production — are becoming the core infrastructure battle for customers and vendors. That framing suggests procurement conversations are shifting from experimentation budgets to ongoing operational economics. (shashi.co)
Running an artificial intelligence model once is an experiment; running it millions of times is a utility bill, and that bill is becoming the fight. (shashi.co) Inference is the repetitive work of serving prompts and generating answers after a model is trained. OpenAI’s public pricing page lists GPT-5.4 at $2.50 per 1 million input tokens and $15 per 1 million output tokens, while GPT-5.4 mini is cheaper at $0.75 input and $4.50 output. (openai.com) Anthropic’s pricing page shows the same split between one-time model choice and recurring usage: Claude Sonnet 4.6 is priced at $3 per million input tokens and $15 per million output tokens, and Claude Haiku 4.5 is $1 input and $5 output. Anthropic also prices cache hits separately at a fraction of standard input cost, which turns prompt reuse into a procurement lever. (platform.claude.com) Cloud vendors are now selling around that operating-cost problem, not just around model quality. Amazon Bedrock says batch inference is 50% cheaper than on-demand, and Bedrock offers Standard, Flex, Priority, and Reserved tiers for different cost and availability trade-offs. (aws.amazon.com) The hardware pitch has shifted the same way. Google Cloud’s pricing page lists Trillium at $2.70 per chip-hour on demand in South Carolina, versus $4.20 for TPU v5p in the same region, and Google says Trillium was built to serve models with lower latency and lower cost. (cloud.google.com, cloud.google.com) Amazon is making a similar argument with its own chips. Amazon Web Services says Trainium2 offers 30% to 40% better price performance than graphics-processing-unit-based Elastic Compute Cloud P5e and P5en instances, and says Trainium3 is designed for “token economics” in agentic and reasoning workloads. (aws.amazon.com) NVIDIA is framing the contest in cost-per-token terms too. In a February 12, 2026 post, NVIDIA said Baseten, DeepInfra, Fireworks AI, and Together AI were cutting inference cost per token by up to 10 times on Blackwell compared with Hopper. (blogs.nvidia.com) That is the backdrop for the latest analyst framing. Shashi Bellamkonda wrote in April 2026 that enterprise demand is moving past pilots, citing Anthropic’s reported jump from more than 500 business customers spending over $1 million annually in February to more than 1,000 by April. (shashi.co) A customer spending $1 million a year on one provider is not buying a demo. At that level, pricing terms like cached input, batch windows, reserved capacity, regional routing, and chip-hour commitments start to look less like product settings and more like line items in an infrastructure contract. (openai.com, platform.claude.com, aws.amazon.com) The next phase of the market is likely to be sold on steadier math than the last one. Training built the models, but inference is where vendors and customers now have to live with the bill every day. (shashi.co, openai.com)
