Understand this story
Databricks announces the general availability of Real-Time Mode for Spark Structured Streaming. This mode processes arriving records one by one with the realTime trigger, reducing end-to-end latency from seconds to milliseconds without rewriting queries; to use it, change the trigger to realTime and adopt the update output mode, with a minimum trigger interval of 5 seconds. This mode progressively covers stateless and stateful queries with Apache Spark 4.1 through 4.3. In benchmarks, stateless transformations had a p99 of 3 milliseconds, while stateful aggregations were about 140 milliseconds; after MakeMyTrip adopted it, P50 dropped from about 1.23 seconds to 44 milliseconds.
AI-generated from public reports
Latest development · Report
Databricks Introduces Spark Real-Time Mode, Reducing Streaming Latency to MillisecondsDevelopments
1 confirmed developments
Databricks announces general availability of Spark Real-Time Mode
Source:Databricks Blog
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 13
Databricks BlogFirst-partySelected
Databricks Introduces Spark Real-Time Mode, Reducing Streaming Latency to Milliseconds
Databricks released the Real-Time Mode trigger for Spark Structured Streaming, which processes arriving records one by one, reducing end-to-end latency from seconds to milliseconds without rewriting queries. To enable it, change the trigger to realTime and use the update output mode, with a minimum trigger interval of 5 seconds; the mode is now generally available on Databricks and, with Apache Spark 4.1 through 4.3, progressively covers stateless and stateful queries. In benchmarks, stateless transformations had a p99 of 3 ms, while stateful aggregations took about 140 ms; after MakeMyTrip adopted it, P50 dropped from about 1.23 seconds to 44 ms.