LiteLLM + Prometheus observability demo
What happened
A new how-to video demonstrates routing LLM traffic through LiteLLM with Prometheus + Grafana for hop-level metrics and Redis for caching/state — and shows alerting patterns that trigger rollbacks or route‑switching on degradation. The demo highlights distributed tracing, latency/error metrics per provider, and using Redis as both cache and short-lived session store for post‑mortems. (youtube.com)
Why it matters
The demo video "LLM Routing in Production: LiteLLM + Prometheus + Grafana + Redis" is hosted on the MLWorks YouTube channel (channel listed at ~2.37K subscribers). (youtube.com)) LiteLLM's proxy exposes a Prometheus /metrics endpoint and documents a prometheus_initialize_budget_metrics setting that runs a cron job every 5 minutes to emit budget metrics for all API keys and teams. (docs.litellm.ai)) For multi-worker LiteLLM deployments the docs require setting PROMETHEUS_MULTIPROC_DIR so Prometheus scrapes aggregate metrics across worker processes instead of per-process shards. (docs.litellm.ai)) LiteLLM's provider budget routing stores spend in Redis, emits a per-provider remaining-budget metric in USD, and supports time-windowed budgets such as "1d" and "30d" to automatically skip providers that exceed their budget. (docs.litellm.ai)) An open-source exporter called "exporter-litellm" on GitHub exposes comprehensive Prometheus metrics for LiteLLM—including usage, cost, performance and operational telemetry—to simplify building alert rules and dashboards. (github.com)) A published Grafana dashboard (ID 24055) for LiteLLM visualizes latency percentiles (p50/p95/p99), token usage, request routes and trace links to help drill into provider-level latency and error spikes. (grafana.com)) Recent repository issues flag operational risks that affect alerting: issue #13644 documents a /metrics access control concern where any existing API key could reach /metrics, and issue #22580 reports a metrics-labeling bug where include_labels is not respected for certain time-to-first-token metrics. (github.com)) Redis is documented as the canonical cache and short-lived session/store for budget and routing state in LiteLLM, and Redis monitoring via redis_exporter or Redis Cloud's Prometheus endpoint is recommended to surface key eviction, memory and latency signals for routing/rollback alerting. (docs.litellm.ai))
Key numbers
- (youtube.com) The demo video "LLM Routing in Production: LiteLLM + Prometheus + Grafana + Redis" is hosted on the MLWorks YouTube channel (channel listed at ~2.37K subscribers).
- (youtube.com)) LiteLLM's proxy exposes a Prometheus /metrics endpoint and documents a prometheus_initialize_budget_metrics setting that runs a cron job every 5 minutes to emit budget metrics for all API keys and teams.
- (docs.litellm.ai)) LiteLLM's provider budget routing stores spend in Redis, emits a per-provider remaining-budget metric in USD, and supports time-windowed budgets such as "1d" and "30d" to automatically skip providers that exceed their budget.
- (github.com)) A published Grafana dashboard (ID 24055) for LiteLLM visualizes latency percentiles (p50/p95/p99), token usage, request routes and trace links to help drill into provider-level latency and error spikes.
Quick answers
What happened in LiteLLM + Prometheus observability demo?
A new how-to video demonstrates routing LLM traffic through LiteLLM with Prometheus + Grafana for hop-level metrics and Redis for caching/state — and shows alerting patterns that trigger rollbacks or route‑switching on degradation. The demo highlights distributed tracing, latency/error metrics per provider, and using Redis as both cache and short-lived session store for post‑mortems. (youtube.com)
Why does LiteLLM + Prometheus observability demo matter?
The demo video "LLM Routing in Production: LiteLLM + Prometheus + Grafana + Redis" is hosted on the MLWorks YouTube channel (channel listed at ~2.37K subscribers). (youtube.com)) LiteLLM's proxy exposes a Prometheus /metrics endpoint and documents a prometheus_initialize_budget_metrics setting that runs a cron job every 5 minutes to emit budget metrics for all API keys and teams. (docs.litellm.ai)) For multi-worker LiteLLM deployments the docs require setting PROMETHEUS_MULTIPROC_DIR so Prometheus scrapes aggregate metrics across worker processes instead of per-process shards. (docs.litellm.ai)) LiteLLM's provider budget routing stores spend in Redis, emits a per-provider remaining-budget metric in USD, and supports time-windowed budgets such as "1d" and "30d" to automatically skip providers that exceed their budget. (docs.litellm.ai)) An open-source exporter called "exporter-litellm" on GitHub exposes comprehensive Prometheus metrics for LiteLLM—including usage, cost, performance and operational telemetry—to simplify building alert rules and dashboards. (github.com)) A published Grafana dashboard (ID 24055) for LiteLLM visualizes latency percentiles (p50/p95/p99), token usage, request routes and trace links to help drill into provider-level latency and error spikes. (grafana.com)) Recent repository issues flag operational risks that affect alerting: issue #13644 documents a /metrics access control concern where any existing API key could reach /metrics, and issue #22580 reports a metrics-labeling bug where include_labels is not respected for certain time-to-first-token metrics. (github.com)) Redis is documented as the canonical cache and short-lived session/store for budget and routing state in LiteLLM, and Redis monitoring via redis_exporter or Redis Cloud's Prometheus endpoint is recommended to surface key eviction, memory and latency signals for routing/rollback alerting. (docs.litellm.ai))
