- Debezium Server and Databricks Zerobus Ingest combine to stream CDC events directly into Unity Catalog Delta tables through a push-based architecture.
- Locked-down private networks that block inbound connections can now securely replicate data outward, bypassing the need for intermediate Kafka message brokers or pull-based connectors.
- Organizations can push raw database changes outward to the lakehouse, drastically simplifying the infrastructure needed to build current-state tables for downstream analytics.
Data in operational databases drives business decisions, and much of that analysis happens in the lakehouse. Getting the data there is hard when the database sits in a private corporate network, a customer-controlled environment or a restricted subnet whose network policies block inbound connections from external services.
Many teams need their PostgreSQL changes in Databricks, but the team that owns the database also wants to own the capture. That team runs the process that reads the write-ahead log and decides which connections leave its network. On the other side, the changes land in governed Delta tables that pipelines and analysts can build on.
Debezium is a common open source tool that teams can own the full deployment to extract a change data capture feed from a database. It sits next to the database, reads its write-ahead log and pushes events outward. Zerobus Ingest is the destination, a push interface that writes records directly into Unity Catalog Delta tables. The open-source Debezium Server Zerobus sink joins the two without the use of a traditional Kafka cluster between capture and the lakehouse.
Engineers at NVIDIA and Databricks teamed up to build this sink and contributed it to the open-source Debezium project. Databricks built the typed Zerobus Ingest sink. Then, NVIDIA contributed envelope mode, which lands the full Debezium change event, along with the delivery safeguards around it, such as idempotency keys and durable acknowledgment before source offsets advance. The Zerobus Ingest sink is shipped in the Debezium Server 3.7 release.
This post covers what Debezium and Zerobus each do, the raw change envelope NVIDIA contributed to the sink, and how landed events become current-state or historical tables. For a hands-on walkthrough, including a live run from a local PostgreSQL table to a current-state table, see the companion how-to blog on the Databricks Community.
The Challenge
Replicating operational database changes into the lakehouse reliably has to solve several problems at once:
- Capture inserts, updates, and deletes reliably. A row that is created, modified, and later removed must produce a faithful stream for each event. This must include deletes, which query based "copy the table" approaches routinely miss.
- Keep the evidence needed for ordering. Downstream logic that reconstructs current state must apply changes in source order, to correctly replicate the source table.
- Avoid overloading the source database. Polling the database on a schedule (
SELECT ... WHERE updated_at > ?) adds query load, scales poorly. - Work with private / customer-controlled networks. When the database sits behind a locked-down perimeter, we know that it can be challenging to connect to these systems and ingest the data.
- Avoid standing up Kafka purely as transport. Kafka is a powerful event bus, but operating a cluster solely to move CDC events from a database into a lakehouse is significant overhead for teams that don't otherwise need it.
- Land the data so that it can be easily processed. The change stream has to arrive in a form the lakehouse can process and query, not in a silo that requires another export step.
Given those requirements, we need to start with the architecture of which data would flow from source to sink.
Overall Target Architecture
In this blog, we’ll explore a traditional message bus architecture compared to an architecture using Zerobus Ingest.
Architecture with a Message bus:
In many enterprise environments, outbound connectivity from inside a perimeter is far easier to permit than inbound access. When your team wants to run a deployment of the change feed extracting agent, a local push model lets data flow outwards to a destination. In a traditional setup, this would be a message bus such as Kafka. Running Kafka requires resources for capacity planning to predict the number of brokers needed to handle peak load for physical scaling and to estimate the number of partitions for logical scaling, which consumers need to keep up with the rate of production. This requires a dedicated team with experience in this Kafka deployment and management.
Once the data lands in Kafka, you have the next job to fulfill, which is to replicate those events to the lakehouse. This could be through a deployment of ingestion jobs such as with Spark, Kafka Connector, or other tooling. The result is a two-hop architecture from Debezium to the lakehouse.
Architecture with Zerobus Ingest:
When your sole destination is the lakehouse, Zerobus allows you to bypass intermediate hops such as the message bus. This both streamlines your end-to-end architecture and eliminates the need to deploy and manage Kafka infrastructure for data transit. When using Zerobus as a sink with Debezium it continues to work during your peak hours, without tuning.
What is Debezium?
Debezium is an open-source, log-based change data capture platform. Rather than polling tables, it reads a database's transaction log, for example, PostgreSQL's write-ahead log (WAL), MySQL's binlog, and MongoDB's change streams. It then emits each change as a structured event. The project began in 2015, and today supports a broad connector ecosystem (PostgreSQL, MySQL, SQL Server, Oracle, MongoDB, and more).
Each change event carries a consistent envelope:
before— the row state before the change (null for inserts)after— the row state after the change (null for deletes)\op— the operation:c(create/insert),u(update),d(delete),r(read/snapshot)source— metadata: database, schema, table, WAL log sequence number (LSN), transaction ID, snapshot flagts_ms— event timestamp
Debezium adds minimal load to the primary database due to its log tailing as opposed to polling to capture changes at low latency, and preserves commit ordering.
Standalone (Debezium Server) vs. the Kafka Connect deployment
Debezium has two operating models. One which runs on Kafka Connect, and the other which is a “stand-alone server” which can be stood up on vanilla compute.
Debezium on Kafka Connect | Debezium Server (standalone) | |
|---|---|---|
Runtime | Plugin inside a Kafka Connect cluster | Standalone Quarkus application |
Native target | Apache Kafka (then sink out to other systems) | Directly to non-Kafka targets via a sink adapter. (Like Zerobus Ingest) |
When to use | Kafka is your canonical event bus; you want Connect's distributed workers, SMTs, schema registry | You don't run Kafka, or you want to push CDC directly to a purpose-built destination |
Debezium Server, the standalone runtime, serves as your source-side agent that tails transaction logs and sends change events directly to Databricks’ Zerobus Ingest, without a Kafka cluster serving as the transport.
The solution: Debezium + Zerobus Ingest
So we have identified Debezium as our log tailer. Now we need to get the events into Databricks.
Zerobus Ingest is a serverless, push-based ingestion service that writes data directly into Unity Catalog–governed Delta tables, with no message broker to run. Producers connect to a Zerobus endpoint and stream records that land in an open-format Delta table. It's built to handle high throughput, highly concurrent workloads. By skipping the step of a message broker, we are able to simplify the architecture, and the maintenance of having to scale and run ingestion pipelines to move the data to the lakehouse.
Put together with Debezium, your CDC pipelines would look something like this:
- Debezium observes the database's log and converts each change into a CDC event.
- Zerobus Ingest transports those events directly into Databricks and writes them into a Delta table — no Kafka cluster as an intermediate transport layer.
- Databricks receives the change stream in a Delta table and can process it into curated tables downstream.
Zerobus Ingest is append-only. Once the change events land, the next step is to build a Type-1 dimensional table by performing a merge to obtain the latest version of each given primary key combination.
NVIDIA’s contribution: envelope mode
The Zerobus sink in Debezium offers two table contracts. Typed mode, the default, maps each change event onto a table with the source’s own columns. Envelope mode, which NVIDIA contributed, lands the full Debezium change event in a fixed nine-column table:
- target_table
- destination
- partition
- operation
- idempotency_key
- key
- value
- source_position
- headers
Records can be encoded as JSON (stored in VARIANT columns) or Protobuf.
Envelope mode keeps the evidence downstream processing needs. source_position carries the source partition and offset, including the PostgreSQL LSN. idempotency_key is a SHA-256 identity of each event, derived from its destination, partition, key, operation and source position, so a redelivered event gets the same key. The key identifies events; it does not order them or deduplicate the table on its own.
If a later group fails, an earlier durable group can be delivered again, so delivery is at-least-once and duplicates are handled downstream with idempotency_key. Envelope mode also filters by destination and operation, and rejects an oversized record before ingestion so its offset is never committed.
Data modelling in Databricks
Once CDC events land in Delta, a common medallion-style flow turns the raw change feed into something analysts and applications can use:
- Bronze: The raw change events, as received. With envelope mode, that is one row per event, carrying its operation, before/after row images, source position (LSN) and
idempotency_key. - Silver: Parsed change records, deduplicated by
idempotency_keyand ordered by source position. Usable representation of the source data, with inserts/updates/deletes interpreted to reconstruct the current state. - Gold: Operational data is transformed with business data for reporting, such as finances, performance metrics, etc.
Two ways to shape the change feed are worth naming briefly:
- Type-1 (current state): collapse the changelog so each key reflects its latest value — a current-state replica of the source table.
- Type-2 (history): preserve each version over time, so you can query the data as it looked at any point.
The companion how-to blog on the Databricks Community walks through this end to end with a local PostgreSQL table. It configures Debezium Server with the Zerobus sink, checks an insert, update and delete in the envelope table, then builds the Type-1 (current-state) table with AUTO CDC, sequencing changes by source LSN.
Conclusion
Choosing the right CDC architecture ultimately depends on your organization's networking constraints, operational preferences, and existing infrastructure.
Requirement / Feature | Debezium + Zerobus (Local Push) | DIY (Debezium + Kafka + Spark) |
|---|---|---|
Ingestion Direction | Push-based (Source agent pushes outward) | Push-based (Source agent pushes outward to Kafka) |
Network Requirements | Requires only outbound access from source network | Requires outbound access to Kafka cluster / brokers |
Private / Customer Networks | Ideal fit for locked-down or private networks | Good fit for private networks, but requires managing Kafka connectivity |
Kafka Infrastructure | Not required (streams directly via Zerobus) | Required (Kafka cluster acting as intermediate buffer) |
Connector Management | Requires running Debezium Server locally | High operational overhead (managing Kafka, Connect, and Spark ingestion jobs) |
Ecosystem Compatibility | Leverages full Debezium connector ecosystem; envelope mode lands raw change events for AUTO CDC silver and gold tables | Leverages full Debezium and Kafka Connect ecosystem with custom Spark streaming to Delta (Bronze) and AUTO CDC jobs |
By combining Debezium Server with Zerobus Ingest, you can deploy a local push architecture that streams changes directly into Delta tables without requiring Kafka as an intermediate message transport or relying on staging areas in cloud object storage, simplifying the end-to-end ingestion path.
This keeps PostgreSQL capture under the control of the team that owns the database while the data lands in Unity Catalog.
Getting started
To build this pipeline yourself, follow the step-by-step guide on the Databricks Community. It runs PostgreSQL changes through Debezium Server and Zerobus Ingest into a Unity Catalog table, then builds a current-state table with AUTO CDC. For Zerobus Ingest setup and its other ingestion interfaces, see the Zerobus Ingest documentation.
FAQs
What is the Debezium Zerobus Ingest sink?
It's a sink in Debezium Server, the standalone Debezium runtime, that sends change events directly to Zerobus Ingest. Zerobus Ingest writes them into Unity Catalog–governed Delta tables, so no Kafka cluster sits between capture and the lakehouse. NVIDIA and Databricks collaborated on the open source contribution.
What is envelope mode?
Envelope mode lands each full Debezium change event in a fixed nine-column table instead of mapping it onto the source's own columns. NVIDIA contributed it to the sink. It keeps the operation, key, value, source position and an idempotency key for downstream processing. Typed mode is the default.
Does the sink deliver each change exactly once?
No, delivery is at-least-once. If a later group of records fails, an earlier durable group can be delivered again. Each event carries an idempotency_key, so you deduplicate on it downstream.
How do I turn the change feed into a current-state table?
Use AUTO CDC to build a Type-1 table that keeps the latest version of each key. Sequence the changes by the source LSN so they apply in source order. To keep every version over time instead, build a Type-2 table.
Get the latest posts in your inbox
Subscribe to our blog and get the latest posts delivered to your inbox.