AI gateway control plane

An AI gateway is turning into the control desk for model traffic, not just a pass-through for prompts and responses. Cloudflare now markets AI Gateway as an “AI application control plane,” and engineer Ranjan Kumar’s six-part series argues teams should run gateways the same way they run API control planes. (workers.cloudflare.com, ranjankumar.in) In plain terms, a gateway is the checkpoint every model call passes through. Because every request crosses that checkpoint, teams can put routing, logging, rate limits, caching, and spend controls in one place instead of scattering them across each app or agent. (truefoundry.com, workers.cloudflare.com) Kumar’s series, published on April 7 and April 8, 2026, breaks that control plane into six jobs: observability, policy, orchestration, versioning, cost governance, and compliance. The argument is that production AI fails less often when those jobs sit at the gateway layer, where operators can change rules without redeploying every downstream service. (ranjankumar.in, ranjankumar.in, workers.cloudflare.com) The first job is observability, which means seeing what the system is doing while it runs. Kumar wrote that one invoice-processing agent burned “several hundred dollars” over a weekend while dashboards stayed green, because the team had traces and logs but could not see the state transitions behind the loop. (ranjankumar.in) Policy is the second job: who can call which model, with what data, under what limits. Kumar made the same case in a January 23, 2026 post about Model Context Protocol, writing that centralized routing and policy let organizations audit every context request instead of coordinating rule changes across “fifteen different teams.” (ranjankumar.in) Orchestration is the third job, and it matters most when one model hands work to another. In Kumar’s April 8 example, a hallucinated stock-keeping unit moved through four downstream systems because the pipeline had retries and circuit breakers, but no halt protocol to stop bad output from spreading. (ranjankumar.in) Versioning is the fourth job: keeping model, prompt, and schema changes from breaking live systems. Kumar described a case where Agent 1 changed an output field from `vendor_name` to `vendor`, and Agent 2 failed silently for six hours because the version combination had not been tested as a matrix. (ranjankumar.in) Cost is the fifth job, and it is moving closer to mainstream infrastructure management. Kumar’s cost-governance piece cites ICONIQ Capital’s 2026 State of AI report saying inference costs run at 23% of revenue for scaling AI-native companies, and it argues per-request token caps miss the bigger problem when routing bugs multiply spend across a fleet. (ranjankumar.in) Compliance is the sixth job, and the calendar is driving urgency. Kumar notes that the European Union AI Act entered into force in August 2024, rules for general-purpose AI models applied on August 2, 2025, and full requirements for high-risk systems take effect on August 2, 2026, with obligations around record-keeping, transparency, oversight, and robustness. (ranjankumar.in) Vendors are already packaging that bundle as product. Cloudflare says its gateway can connect to multiple model providers, route requests by latency, cost, or availability, and expose logs, token usage, request status, and cost from one dashboard and one bill. (workers.cloudflare.com) The thread running through all six parts is simple: if AI calls are now production traffic, the gateway becomes the place where teams see them, govern them, price them, and shut them down when they go wrong. That is the same job API control planes took on for microservices, now rewritten for models and agents. (truefoundry.com, workers.cloudflare.com, ranjankumar.in)