Skip to main content
  • Databricks Real-Time Mode is a Spark Structured Streaming trigger that processes records as they arrive, for millisecond, end-to-end latency.
  • You enable it by changing one trigger line. The source, transformations and checkpoint stay the same, so no rewrite is needed.
  • Use it for fraud detection, personalization and real-time AI features, where Spark now covers latency that used to require Apache Flink.

Real-Time Mode is a trigger for Apache Spark™ Structured Streaming that processes each record as soon as it arrives instead of waiting for the next micro-batch. It brings Spark streaming from seconds down to milliseconds of end-to-end latency, on the same engine and with the same APIs you already use for batch and micro-batch streaming.

Before Real-Time Mode, teams that needed sub-second responses often added a second, stream-only engine. Real-Time Mode removes that requirement for most workloads. You change the trigger on an existing query, and Spark runs it continuously.

How does Real-Time Mode work?

Real-Time Mode replaces micro-batch scheduling with long-running tasks that keep reading and processing data. A micro-batch query waits for a batch boundary, schedules tasks and runs a blocking shuffle between stages, which sets a latency floor of roughly a second. Real-Time Mode removes that overhead with three changes, explained in depth in Breaking the microbatch barrier: the architecture of Apache Spark Real-Time Mode.

Diagram of the three Real-Time Mode innovations: continuous data flow, pipeline scheduling of all stages at once, and a non-blocking streaming shuffle between stages.
Figure 1. Real-Time Mode combines continuous data flow, pipeline scheduling and a streaming shuffle so records move from source to sink without waiting for batch boundaries.
Timeline comparing Spark micro-batch execution with gaps between batches to Real-Time Mode, where long-running tasks process records continuously between checkpoints.
Figure 2. Micro-batch waits for batch boundaries and blocking shuffles. Real-Time Mode runs stages continuously and checkpoints in the background.

The trigger interval controls how often Spark commits a checkpoint and reports progress. It doesn't control how long a record waits. Records flow through the long-running task the whole time, and checkpoints commit in the background.

How do I turn on Real-Time Mode?

Change the trigger on a Structured Streaming query to realTime and set update output mode. The rest of the query stays the same.

The minimum trigger interval is 5 seconds, and five minutes is a common production setting. On Databricks, enable Real-Time Mode on classic compute running Databricks Runtime 16.4 LTS or later, with Photon, autoscaling and spot instances turned off, and set spark.databricks.streaming.realTimeMode.enabled to true. You can also run it on serverless compute inside Lakeflow Spark Declarative Pipelines in continuous mode.

Size for every stage at once. Because all stages run concurrently, the cluster needs a task slot for every task in the query. A Kafka source with 8 partitions plus one shuffle with 20 partitions needs 28 slots.

What does Real-Time Mode support?

Real-Time Mode supports Kafka-based sources and sinks, row-level foreach sinks and most streaming operators, including stateful ones. This table summarizes the current support on Databricks. See the Real-Time Mode reference for the complete list.

AreaSupportedNot supported
SourcesApache Kafka, AWS MSK, Amazon Kinesis, Event Hubs via the Kafka connectorDelta tables as a source
SinksKafka, foreach (ForeachWriter)Delta tables, foreachBatch
Output modeUpdateAppend, complete
Stateless operatorsProjections, filters, UDFs, stream-static joinsNone of note
Stateful operatorsAggregations, tumbling and sliding windows, deduplication, transformWithState (row-based)Session windows, transformWithStateInPandas, event-time timers
Compute (Databricks)Classic compute, Lakeflow pipelines on classic or serverlessServerless notebooks and jobs, Free Edition

Table 1. Real-Time Mode support on Databricks. Source: Databricks Real-Time Mode reference.

To land Real-Time Mode results in a Delta table, write them to Kafka and run a separate micro-batch query from Kafka to Delta.

Which Spark version supports Real-Time Mode?

Real-Time Mode is generally available on Databricks and is rolling out across open source Apache Spark releases. On Databricks, it has been generally available since March 19, 2026, including stateful operators and transformWithState.

ReleaseWhat it added
Apache Spark 4.1Real-Time Mode for stateless Scala queries
Apache Spark 4.2PySpark Real-Time Mode trigger
Apache Spark 4.3Stateful operators: aggregation, deduplication and transformWithState
Databricks (GA, March 2026)Stateless and stateful queries, initial state loading, Lakeflow pipeline support

Table 2. Real-Time Mode availability. Sources: Spark 4.1.0 release notes, Apache Spark Real-Time Mode docs.

How fast is Real-Time Mode?

Real-Time Mode reaches single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries in published benchmarks. Actual latency depends on the source, operators, state size, sink and cluster capacity.

WorkloadPublished p99 latencySource
Stateless transformation3 msDatabricks latency benchmark
Stream-static join9 msDatabricks latency benchmark
Stateful aggregation (feature computation)14 msDatabricks latency benchmark
dropDuplicates136 msDatabricks stateful benchmark
Window aggregation139 msDatabricks stateful benchmark
transformWithState141 msDatabricks stateful benchmark

Table 3. Published Real-Time Mode latency results. Sources: latency-benchmarks repository and the Real-Time Mode tutorial.

To monitor a running query, use the Real-Time Mode latency metrics, not batch duration. Batch-duration metrics don't reflect end-to-end latency, and the reported end-to-end metric excludes sink write time. See optimize and monitor Real-Time Mode performance.

In production, MakeMyTrip cut P50 latency from about 1.23 seconds to 44 ms and P99 from more than a minute to about 500 ms, which drove a 7% lift in click-through rates.

Bar chart showing MakeMyTrip P50 latency falling from about 1.23 seconds to 44 ms and P99 from over a minute to about 500 ms with Spark Real-Time Mode.
Figure 3. MakeMyTrip cut P50 latency from about 1.23 seconds to 44 ms and P99 from over a minute to about 500 ms after adopting Real-Time Mode.
REPORT

The agentic AI playbook for the enterprise

Read now
ai

How do I use state in Real-Time Mode?

Use the standard streaming operators for aggregation and deduplication, and use transformWithState for custom per-key logic. State lives inside the long-running task, so stateful work stays on the low-latency path. Set a watermark so Spark can release old state.

For custom stateful logic in Python, use the row-based transformWithState API, which requires Databricks Runtime 17.3 LTS or later. The pandas variant, transformWithStateInPandas, isn't supported in Real-Time Mode. See build a custom stateful application.

When should I use Real-Time Mode?

Use Real-Time Mode when the business action has to happen within a second, and use a cheaper trigger when it doesn't. Real-Time Mode runs continuously on dedicated compute, so its cost is steady.

  • Fraud and risk scoring. DraftKings uses Real-Time Mode with transformWithState for fraud detection in live sports betting.
  • Personalization and recommendations. MakeMyTrip serves personalized hotel results with 44 ms P50 latency.
  • Real-time AI features and agent context. The same pipeline computes features for model training and online inference, and keeps AI agents supplied with the latest events.
  • Operational alerting and IoT monitoring. Sensor and security signals trigger actions in milliseconds.

When seconds or minutes are acceptable, micro-batch or the availableNow trigger is the better choice and costs less. Because the code is the same, you can start with a cheaper trigger and switch later.

Is Real-Time Mode an alternative to Apache Flink?

Yes, for most workloads. Real-Time Mode gives Spark the millisecond latency that once made Apache Flink the default for operational streaming, while keeping one engine for batch, SQL and machine learning. In the Databricks benchmark, Real-Time Mode matched Flink on stateless work and had lower p99 latency on stateful aggregation and stream-static joins, which Databricks summarized as up to 92% faster. For a full comparison, read Apache Spark vs Apache Flink.

Real-Time Mode on Databricks

Databricks was founded by the original creators of Apache Spark and runs Real-Time Mode as a generally available feature. On top of open source Spark, it adds:

  • Stateful Real-Time Mode today: aggregations, deduplication and transformWithState with initial state loading
  • Lakeflow Spark Declarative Pipelines: real-time flows on classic or serverless compute
  • Unity Catalog: governance and lineage for real-time and batch tables

Code written for open source Spark runs on Databricks unchanged. To try it, follow the Apache Spark tutorial: Real-Time Mode.

FAQs

What is Real-Time Mode in Spark?

Databricks Real-Time Mode is a Structured Streaming trigger that processes records continuously as they arrive instead of in micro-batches. It brings Spark streaming down to millisecond, end-to-end latency without a code rewrite.

Is Real-Time Mode the same as continuous processing?

No. Continuous processing was an earlier experimental mode with limited operator support. Real-Time Mode is a new design with long-running tasks, pipeline scheduling and a streaming shuffle, and it supports stateful operators.

Does Real-Time Mode support stateful operations?

Yes. On Databricks, Databricks Real-Time Mode supports aggregations, deduplication and transformWithState today. In open source Apache Spark, stateful support arrives with Spark 4.3.

Can Real-Time Mode write to Delta Lake?

Not directly. Real-Time Mode writes to Kafka or a foreach sink. To land results in a Delta table, run a separate micro-batch query from Kafka to Delta.

Does Real-Time Mode run on serverless or Free Edition?

Databricks Real-Time Mode runs on classic compute and in Lakeflow Spark Declarative Pipelines, including serverless pipelines. It doesn't run in serverless notebooks or jobs, so it isn't available in Databricks Free Edition notebooks.

Get started with Real-Time Mode on Databricks

Get the latest posts in your inbox

Subscribe to our blog and get the latest posts delivered to your inbox.

Sign up

View all blogs slice-start id="_gatsby-scripts-1" slice-end id="_gatsby-scripts-1"