AI Trends

Databricks Opens Genie ZeroOps Pipeline Agent for Public Beta

Unite.AIUpdated Nov 20 06:27

AI summary

On November 19, 2026, Databricks opened the Genie ZeroOps public beta. This data pipeline agent can automatically detect failures, investigate their causes, and prepare fixes. It can aggregate logs, source code, table metadata, and lineage information, use SQL to identify root causes, and validate fixes in an isolated sandbox through unit tests and dry runs. ZeroOps will not modify production code or data without approval, but it will continuously use large language models to process selected assets. Its current monitoring scope covers jobs, pipelines, and tables.

Why read this

AI score 69

The public beta demonstrates the complete workflow from failure detection to controlled remediation, while also clearly outlining the enterprise deployment trade-offs involving continuous AI processing, shared permissions, and external notifications.

Topics

Tags

Body

Databricks opened Genie ZeroOps to public beta on November 19, 2026, after private-preview testing of its autonomous data-pipeline agent. The company says ZeroOps observed more than 80,000 real customer issues during that limited preview.

ZeroOps detects failures, investigates their causes and prepares fixes. The beta adds shared inboxes, notifications that summarize each issue and recommendation, custom instructions for company-specific workflows, and one-click pull-request creation for Git-backed jobs.

Private-preview testing covered hundreds of companies, and Databricks reports that more than half of the issues required organization-specific knowledge to triage, rank and resolve properly. That finding drove features that let teams identify known flaky sources, preferred fixes and naming conventions, or filter expected failures and lower-priority notifications.

From Detection to Reviewed Fixes

ZeroOps watches for explicit job and pipeline failures as well as tables that stop updating or arrive with unexpectedly few rows, problems that may not trigger a conventional job failure. It gathers logs, source code, table metadata and lineage, runs SQL queries to identify root causes, and prioritizes findings by downstream impact. Proposed fixes are tested with unit tests and dry runs in an isolated sandbox before anyone applies them.

Investigations start automatically when an issue is detected. Built-in access to run history, data-quality signals and Unity Catalog lineage, which maps dependencies between assets, connects code diagnostics to the state of the data. Alerts can go to Slack, Microsoft Teams, PagerDuty or generic webhooks; email destinations are unsupported.

Approval and Data Controls

ZeroOps does not change production code or data without approval. The documented guardrails require an explicit user instruction and confirmation before it pushes a Git-backed fix; the push adds a commit without force-pushing or rewriting branch history. Naming a repository’s default branch can bypass pull-request review unless branch protection blocks direct pushes.

Databricks says the agent uses Anthropic and OpenAI models on Databricks-hosted endpoints inside its security perimeter. Selecting assets authorizes continuous, autonomous large-language-model processing until the feature is disabled, including repository contents for Git-backed jobs. Before an external notification includes a finding’s title or summary, the agent checks for sensitive information such as names and email addresses and masks what it finds. Copies stored in outside systems fall outside Databricks’ retention and deletion controls.

A configuration can be shared with up to 25 individual workspace users, but not with groups or service principals. Collaborators can work on its findings, yet shared users see code excerpts, suggested fixes and sample data rows even without access to the underlying assets. Sharing does not grant access to those assets.

Setup Requirements and Beta Scope

An administrator must enable Genie ZeroOps from the workspace Previews page. Users then select assets and need OWNER, CAN MANAGE or CAN VIEW permission on them. The setup process backfills findings from the previous 24 hours before reporting new issues. Investigating a Git repository requires a credential with read access; creating a pull-request branch requires write access.

Databricks introduced ZeroOps on June 16, 2026 with planned private-preview support for jobs, pipelines, tables and machine-learning workloads. Current documentation states that monitoring covers jobs, pipelines and tables, not models or apps, and that performance and cost regressions are outside its detection scope. The beta operates at workspace rather than account level, is not HIPAA compliant, and requires separately billed data-quality monitoring to detect late-arriving or incomplete table data.

ZeroOps automates investigation and fix preparation while leaving production authority with the customer. The beta’s central enterprise trade-off is less manual diagnosis in exchange for continuous AI processing of selected operational data, making sharing visibility, external notifications and Git review part of deployment rather than separate administrative details.

Source

Unite.AI