Tools for LLM gateway observability

An LLM gateway is becoming a new control point in artificial intelligence software: one layer decides which model handles a request, records what happened, and tracks the bill. (medium.com) That layer sits between an app and model providers such as OpenAI or Anthropic, much like an application programming interface gateway sits in front of ordinary web services. Whatnot said its platform uses shared middleware and provider routing so product teams do not wire each model provider separately. (medium.com) The basic job is routing and record-keeping. A gateway can send a simple prompt to a cheaper model, fall back when one provider fails, and log tokens, latency, outputs, and errors for each call. (arxiv.org) (openai.github.io) A second layer of tools is growing around that gateway. Langfuse, an open-source project, says it combines tracing, prompt management, evaluations, and analytics dashboards so teams can see quality, cost, and latency in one place. (langfuse.com 1) (langfuse.com 2) Prompt management is one piece of that shift. Langfuse says teams can store prompts centrally, version them, and update them outside application code instead of leaving instructions hardcoded across services. (langfuse.com) Tracing is another piece. OpenAI’s current documentation says its Agents software development kit records model calls, tool calls, handoffs, guardrails, and custom events, giving developers a run-by-run trace they can inspect in development or production. (developers.openai.com) (openai.github.io) What platform teams want from these systems is not just uptime. Langfuse says its dashboards expose cost, latency, and evaluation scores, while Whatnot said its internal platform work is organized around velocity, reliability, and trust. (langfuse.com) (medium.com) The pressure behind the tooling is economic as much as technical. A February 2025 paper on model routing described the goal as sending each prompt to the smallest feasible model, cutting inference cost without giving up needed quality. (arxiv.org) Open-source projects are also multiplying around the same problem. OmniRoute describes itself as an OpenAI-compatible gateway with routing, retries, fallbacks, rate limits, caching, and observability, and OpenLIT pitches open-source observability built on OpenTelemetry-style tracing and metrics. (github.com 1) (github.com 2) The result is a new kind of software plumbing for artificial intelligence products. As more companies use several models at once, the question is shifting from which model is best to which system can show, request by request, what the model did and what it cost. (medium.com) (langfuse.com)