Gateways are now the LLM control plane

Right now, many companies reach language models through a mess of direct connections, one app talking to OpenAI, another to Anthropic, another to an internal model, with billing, logging, and safety rules scattered across all three. MuleSoft said on March 30, 2026 that it wants one governed layer for every large language model call instead, using its AI Gateway as a single control plane. (mulesoft.com) A gateway is the checkpoint in front of a system, like a building lobby where every visitor shows ID before going upstairs. In software, an API gateway already sits in front of application programming interfaces, and MuleSoft says the same pattern now needs to sit in front of model calls too. (mulesoft.com) The pitch is simple: one endpoint for developers, many model providers behind it. MuleSoft’s product page says teams can connect to any provider through a single endpoint, route requests intelligently, and track spend across enterprise large language model usage. (mulesoft.com) That solves three concrete problems that show up fast in production. The first is policy, because a company may want one rule for redacting personal data, one rule for blocking certain prompts, and one rule for which team can use which model, instead of rebuilding those checks inside every app. (mulesoft.com; learn.microsoft.com) The second is access, because model providers all authenticate differently and expose slightly different interfaces. Microsoft’s AI gateway docs describe the same need in Azure API Management: authenticate and authorize access, import OpenAI-compatible endpoints, and expose a wide range of model and agent backends through one managed layer. (learn.microsoft.com) The third is visibility, because finance and security teams want one place to see token usage, quotas, logs, and failures. MuleSoft says its gateway adds full token usage accountability, while Databricks describes AI Gateway as a control plane for analyzing usage, configuring permissions, and managing capacity across providers. (mulesoft.com; learn.microsoft.com) This is why “gateway” is starting to sound like “control plane.” In infrastructure, a control plane is the layer where administrators set rules and observe the system, and vendors from MuleSoft to Microsoft to Databricks are now describing AI gateways in exactly those terms. (mulesoft.com); (learn.microsoft.com); (learn.microsoft.com) The story gets bigger when the traffic is not just chat prompts from one app but chains of tools and agents calling each other. MuleSoft’s March 30 announcement says its large language model gateway now sits alongside governance for Model Context Protocol and Agent-to-Agent traffic, which means the checkpoint is moving from single prompts to whole agent workflows. (mulesoft.com) That is also why open-design discussions are shifting from “proxy a model” to “govern an agent system.” The emerging design is a layer that can inspect requests, apply policy, route to different providers, and log what happened across models, tools, and agent calls, not just forward text to one chatbot. (mulesoft.com; developer.konghq.com; aws.amazon.com) The reason vendors are pushing this now is that companies already have multi-model sprawl. MuleSoft cites a Gartner forecast that 70 percent of organizations building multi-large-language-model applications will use AI gateway capabilities by 2027, up from less than 5 percent in 2024. (mulesoft.com) So the shift is not that gateways suddenly appeared in 2026. The shift is that the old API gateway job, enforce rules at one choke point, is being extended to language models, model context servers, and software agents, and vendors now want that choke point to be the operating layer for enterprise artificial intelligence. (mulesoft.com; learn.microsoft.com)