Databricks ships Spark RTM GA
Databricks announced general availability of Real‑Time Mode for Spark Structured Streaming, claiming millisecond‑level latencies and citing customers who cut end‑to‑end latency by 80%+ while computing 250+ ML features. The move promises a single engine path for ultra‑low‑latency streaming workloads that previously needed separate runtimes. (x.com)
Databricks published the General Availability announcement for Real‑Time Mode on March 19, 2026, in a blog post authored by Navneeth Nair and Giselle Goicochea. (databricks.com) Apache Spark 4.1 includes the Real‑Time Mode SPIP in its release notes, marking RTM as an upstream Spark feature rather than a Databricks‑only extension. (spark.apache.org) Databricks describes three core architectural changes behind RTM — continuous data flow, pipeline scheduling (scheduling all stages concurrently), and a streaming shuffle layer — as the levers that change Structured Streaming’s execution model. (databricks.com) The RTM execution model switches Structured Streaming from blocking microbatches to a hybrid, non‑blocking approach that runs long‑lived batches and schedules stages concurrently to reduce per‑record latency. (databricks.com) Operational guidance in the docs notes RTM’s default long‑running batch duration is five minutes and requires cluster task slot provisioning so available task slots are at least equal to the number of tasks across stages; Databricks documents RTM availability starting in Databricks Runtime 16.4 LTS and above. (docs.databricks.com) Databricks has published a performance analysis positioning RTM ahead of Apache Flink for many low‑latency feature‑engineering workloads, and the company first exposed RTM to customers in a public preview that began in August 2025. (databricks.com) Product docs and tutorials include concrete examples for common sources and sinks (including Kafka) and step‑by‑step “get started” guides for configuring real‑time clusters and observability for production rollouts. (learn.microsoft.com)
