MLflow adds an AI Gateway
MLflow introduced an AI Gateway that lets teams expose a single, governed endpoint routed across 15+ LLM providers while adding evals and cost controls. The feature aims to simplify production ML workflows by centralizing model access and governance without extra runtime dependencies. That can reduce integration friction when your app needs to switch or multi-source models in production. (x.com)
Most teams do not call just one language model anymore. A chatbot might use OpenAI for one task, Anthropic for another, and Amazon Bedrock as a backup, which turns one app into three software integrations and three sets of keys to manage. (mlflow.org) An artificial intelligence gateway is the layer that sits in front of those model companies like a hotel front desk. Your app talks to one address, and the gateway decides which provider actually gets the request. (mlflow.org) That setup fixes a boring but expensive problem: direct model calls scatter provider keys across notebooks, continuous integration systems, and developer laptops. MLflow’s own launch post describes teams losing track of token usage, request logs, and where sensitive data is being sent. (mlflow.org) MLflow is the open-source tool many machine learning teams already use to track experiments, models, traces, and evaluations. Instead of making companies bolt on a separate gateway product, MLflow has now folded that gateway into the same system. (mlflow.org, mlflow.org) The new piece is called MLflow AI Gateway, and MLflow says it gives teams one secure endpoint for every large language model provider they use. The same docs say it supports routing, traffic splitting for A/B tests, and automatic failover chains when one provider has an outage. (mlflow.org) MLflow also made a design choice that will matter to engineers already running it in production. In the current release line, the gateway runs inside the MLflow Tracking Server instead of as a separate process, which removes another service to deploy and monitor. (github.com, mlflow.org) The compatibility trick is that apps can use an OpenAI-compatible software development kit and just point the base URL at the gateway. MLflow’s product page says the app then uses the gateway endpoint name as the “model,” which means less code changes when a team swaps providers behind the scenes. (mlflow.org) MLflow is pitching governance as much as convenience. Its docs say the gateway centralizes credentials, tracks usage and costs, logs requests for audit trails, and can enforce controls such as personally identifiable information redaction and access policies. (mlflow.org, mlflow.org) That matters because model choice is turning into a moving target. When one provider raises prices, adds a better model, or goes down for an hour, the company with a gateway can change routing in one place instead of editing every application that calls the model directly. (mlflow.org) MLflow’s own positioning goes one step further: the gateway sits next to tracing and evaluation, so the same platform that sends the request can also measure the response. In practice, that means a team can compare cost, latency, and answer quality across providers without stitching together separate tools first. (mlflow.org, docs.databricks.com) The bigger shift here is that “pick a model” is starting to look less like a one-time architecture decision and more like internet routing. MLflow is betting that companies want one control tower for model access, not a pile of hard-coded provider connections buried across production apps. (mlflow.org, mlflow.org)
