AI gateway control planes

AI gateways are turning into a lightweight control plane for model traffic, giving platform teams one place to route, secure, and watch large language model calls. (aigateway.envoyproxy.io) In Kubernetes terms, the control layer usually defines policy and routes, while the data layer handles live requests. Gateway API splits those jobs across resources such as GatewayClass and HTTPRoute, with a controller managing the gateway and routes attaching application traffic rules. (kubernetes.io) HTTPRoute is now a standard Gateway API resource for describing how HTTP requests move from a gateway listener to backends. That gives AI teams a familiar way to steer prompts and responses without inventing a separate routing model for every provider. (gateway-api.sigs.k8s.io) The pattern is showing up in AI-specific gateway projects that sit on top of that Kubernetes plumbing. Envoy AI Gateway says it uses Envoy Gateway as the control plane and Envoy Proxy as the data plane, and it generates Gateway API resources such as HTTPRoute from its own AI route objects. (aigateway.envoyproxy.io; aigateway.envoyproxy.io) NGINX is framing a similar split for general gateway infrastructure. NGINX Gateway Fabric describes itself as a Gateway API implementation powered by NGINX, and its documentation separates control-plane configuration from the request-serving data plane. (kubernetes.nginx.org; docs.nginx.com) The operational pitch is centralization. A single gateway endpoint can hide provider-specific APIs from application teams while platform teams set authentication, rate limits, and top-level routing in one layer before traffic reaches a model service. (github.com; awslabs.github.io) Observability is a second reason these gateways are being treated like control planes instead of simple reverse proxies. Envoy AI Gateway documents token, latency, and model-performance metrics through OpenTelemetry’s generative artificial intelligence conventions, plus tracing tied to OpenInference conventions for multi-step model requests. (aigateway.envoyproxy.io; opentelemetry.io; arize-ai.github.io) That fits the way AI traffic behaves in production. One user request can trigger several model calls, tool invocations, and retrieval steps, and OpenInference defines traces as the full execution path across those spans rather than a single API hit. (arize-ai.github.io) The gateway model also lines up with where Kubernetes networking has moved over the last two years. Kubernetes said in its May 9, 2024 v1.1 release post that Gateway API had already reached general availability in October 2023, and the project has kept expanding standardized routing features since then. (kubernetes.io; gateway-api.sigs.k8s.io) The result is that “AI gateway” increasingly means a policy and observability layer, not just a proxy in front of an application programming interface. The teams building these stacks are borrowing the same control-plane and data-plane split already used in modern Kubernetes gateways, then applying it to model routing. (aigateway.envoyproxy.io; kubernetes.io)