Databricks adds OpenTelemetry tracing

Databricks said on May 22 that it had added production-ready OpenTelemetry tracing to Unity Catalog, extending its governance layer into the telemetry generated by AI agents and applications. The company said customers can store traces in Unity Catalog tables, query them with SQL and use the data for analytics, evaluation and monitoring. Databricks framed the change as a way to handle the volume of trace data produced by agentic systems while keeping it under the same access controls used for other data and AI assets. ### Where does the new tracing actually live? Unity Catalog is the storage and governance layer Databricks is using for the new tracing workflow. Databricks’ documentation says OpenTelemetry traces can be stored in Unity Catalog tables rather than only in the MLflow control plane service, which shifts access control to Unity Catalog schema and table permissions. That means users with access to those tables can view traces across experiments stored there, according to the company. (databricks.com) Delta tables are the underlying format for that storage. Databricks said customers can keep large volumes of traces for long-term retention and analysis, then query the data directly through Databricks SQL warehouses for reporting and investigation. ### Why is Databricks tying this to OpenTelemetry? OpenTelemetry is the standard Databricks is using for compatibility with external tools and clients. (docs.databricks.com) In its May 22 blog post, Databricks said traces in Unity Catalog create what it called a “continuous improvement flywheel” for AI agents by linking observability with analytics, evaluations and monitoring. The company’s documentation also says OTel-formatted traces improve compatibility with outside systems and use URI-based trace IDs instead of Databricks’ earlier experiment-oriented format. April 9 documentation shows Databricks had already exposed an OTLP endpoint through its Zerobus Ingest service. That endpoint lets customers push traces, logs and metrics directly into Unity Catalog Delta tables using standard OpenTelemetry SDKs and collectors, without custom libraries, according to the company. ### What does this change for teams running AI agents? Databricks said the feature is aimed at “any agent, anywhere,” including production AI systems that span several services. (databricks.com) The company said tracing data in Unity Catalog can be used to understand agent behavior, support debugging and connect observability data to downstream monitoring and evaluation workflows. (docs.databricks.com) Databricks documentation gives more concrete uses. A guide for Databricks Apps, updated May 12, says app telemetry can automatically capture system logs, traces and metrics, including usage events such as user logins and direct API requests, and persist them to Unity Catalog tables using OpenTelemetry. Separate documentation for custom model serving says persisted telemetry can be used for root-cause analysis, endpoint health monitoring and compliance queries with SQL. (databricks.com) ### Is this broadly available yet? Public Preview is the current status for several parts of the rollout. Databricks’ Unity Catalog trace storage documentation, last updated May 12, labels the feature as Public Preview and says workspace administrators can control access from the previews page. Databricks Apps telemetry is also marked Public Preview, while the OTLP client configuration for Zerobus Ingest is marked Beta. (docs.databricks.com) Unity Catalog-enabled workspaces are a prerequisite. Databricks’ setup pages say customers need a Unity Catalog-enabled workspace and must configure target tables, permissions and, in some cases, service principals before sending telemetry data. ### What should readers watch next? May 12 and April 9 documentation updates show Databricks is still expanding the plumbing around the feature, including storage, querying and ingestion paths. (docs.databricks.com) The company has already published setup guides for storing traces, querying them in Databricks SQL and exporting third-party traces from Langfuse into Databricks MLflow for storage in Unity Catalog. Databricks’ next visible milestones are likely to come through those documentation pages and product status labels. As of May 23, the main blog announcement is dated May 22, the Unity Catalog trace storage and Databricks Apps telemetry pages were updated May 12, and the OTLP ingestion configuration page was updated April 9. (databricks.com) (docs.databricks.com)