Understand this story
On November 9, 2026, Databricks released the Beta version of UCDataset on AI Runtime. According to the Databricks Blog, this is a native PyTorch data loader that can stream Unity Catalog tables directly to each GPU in a training job, without the need to export or copy data. In a simulated production test that streamed a 300 GB feature table to train a recommendation model on 8 NVIDIA H100 GPUs, training time was reduced by an average of 10x compared with reading Parquet copies directly. The package is now available on PyPI and can be installed via pip install databricks-uc-dataset, and it automatically handles multi-GPU data sharding and Unity Catalog permissions. There are currently no further developments.
AI-generated from public reports
Latest development · Report
Databricks Launches UCDataset: Direct Connection from Unity Catalog Tables to GPUs, 10x Faster TrainingDevelopments
1 confirmed developments
- Source:Databricks Blog
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 9
Databricks BlogFirst-party
Databricks Launches UCDataset: Direct Connection from Unity Catalog Tables to GPUs, 10x Faster Training
Databricks has released the Beta version of UCDataset on AI Runtime, a native PyTorch data loader that can stream Unity Catalog tables directly to each GPU in training jobs without exporting or copying data. In a simulated production test training a recommendation model on 8 NVIDIA H100 GPUs by streaming a 300 GB feature table, training time was reduced by an average of 10x compared to reading Parquet copies directly. The package is now available on PyPI and can be installed via pip install databricks-uc-dataset, and it automatically handles multi-GPU data sharding and Unity Catalog permissions.