AI News

Trending story

Databricks announces general availability of Spark Real-Time Mode

1 reports1 story updatesUpdated Nov 13 15:49

Understand this story

AI overview

Databricks announces the general availability of Real-Time Mode for Spark Structured Streaming. This mode processes arriving records one by one with the realTime trigger, reducing end-to-end latency from seconds to milliseconds without rewriting queries; to use it, change the trigger to realTime and adopt the update output mode, with a minimum trigger interval of 5 seconds. This mode progressively covers stateless and stateful queries with Apache Spark 4.1 through 4.3. In benchmarks, stateless transformations had a p99 of 3 milliseconds, while stateful aggregations were about 140 milliseconds; after MakeMyTrip adopted it, P50 dropped from about 1.23 seconds to 44 milliseconds.

AI-generated from public reports

Latest development · Report

Databricks Introduces Spark Real-Time Mode, Reducing Streaming Latency to Milliseconds

Developments

1 confirmed developments

  1. Databricks announces general availability of Spark Real-Time Mode

    Source:Databricks Blog

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 13

  1. Databricks BlogFirst-partySelected

    Databricks Introduces Spark Real-Time Mode, Reducing Streaming Latency to Milliseconds

    Databricks released the Real-Time Mode trigger for Spark Structured Streaming, which processes arriving records one by one, reducing end-to-end latency from seconds to milliseconds without rewriting queries. To enable it, change the trigger to realTime and use the update output mode, with a minimum trigger interval of 5 seconds; the mode is now generally available on Databricks and, with Apache Spark 4.1 through 4.3, progressively covers stateless and stateful queries. In benchmarks, stateless transformations had a p99 of 3 ms, while stateful aggregations took about 140 ms; after MakeMyTrip adopted it, P50 dropped from about 1.23 seconds to 44 ms.

Related stories

  1. Related storiesBaidu Wenxin releases ERNIE 5.5 Flash Preview
  2. Related storiesBaidu launches large-model service trading platform "Ciyuansuo"
  3. Related storiesDatabricks launches Lakehouse//RT real-time analytics warehouse
  4. Related storiesDatabricks releases new version of AI Agent Genie