Skip to main content
  • Tabular foundation models can make predictions on new datasets without training a task-specific model, speeding up predictive ML experiments.
  • Compare tabular foundation models with existing baselines on accuracy, inference latency, GPU memory use and cost before choosing a model.
  • Databricks AI Runtime provides managed GPUs, MLflow tracking and Unity Catalog integration for experimenting with and fine-tuning TFMs.

Many of the most valuable predictive machine learning applications rely on tabular data. From credit risk and customer churn to demand forecasting and predictive maintenance, these use cases have traditionally required teams to develop, train and maintain models for each prediction task. Tabular foundation models (TFMs) offer another approach: pretrained models that can make predictions on new datasets without task-specific training.

Productionizing ML systems at scale remains a challenge for many enterprises for the following reasons:

  1. Model development: Each use case requires data preparation, feature engineering, model selection, training and evaluation. Finding a model that meets accuracy and performance requirements can take substantial experimentation.
  2. Model maintenance: Once deployed, models require monitoring for performance degradation and data drift. Teams may also need to maintain retraining pipelines as datasets and business requirements change.

Recently, agentic solutions have helped accelerate these expensive operations, but they still rely on a bespoke-trained model for each ML task or dataset. To remove this dependency, pioneering research labs have applied the transformer architecture to tabular data at scale, creating a new category of models called Tabular Foundation Models (TFMs). Because TFMs come pretrained, they require no training on your dataset, meaning bespoke training and retraining pipelines largely disappear.

Despite the impressive accuracy these models demonstrate on public benchmarks, they have not yet cracked into the mainstream of enterprise AI. This is partially due to the nascent nature of the field, as not many practitioners know about them yet, but also due to compute infrastructure requirements. 

In this article, we'll examine how TFMs work, compare recent models and explain the trade-offs involved in adopting them for enterprise ML. We'll then demonstrate how Databricks AI Runtime supports TFM experimentation and fine-tuning with managed GPU infrastructure, integrated development tools and model governance.

How tabular foundation models work

TFMs largely eliminate the need for training and the overhead of maintaining bespoke models. They are pretrained on millions of diverse synthetic datasets so that they can infer relationships between variables in an unseen dataset. At prediction time, the labeled rows are passed as context alongside the rows to predict. Attention, the core of the transformer architecture, then relates each new row to those examples. Predictions are produced by a single forward pass, a process called in-context learning (ICL).

While this zero-shot approach provides a powerful baseline out of the box, TFMs can also be fine-tuned. By updating the model's pretrained weights on a specific dataset, ML teams can push predictive accuracy even higher for their most critical use cases.

How transformers process tabular data

The main complexity for transformers in TFMs is handling various input types. Unlike words in text or pixels in an image, a table mixes numerical, categorical, and boolean features, each with its own scale and meaning. Efficiently embedding these inputs into a common latent space has been a major challenge. The community solved this by developing specialized tokenizers that map both categorical and numerical features into a single, shared representation.

Once an efficient tokenization for every cell had been achieved, the problem was how to effectively learn the relationship between rows and columns. TabPFNv2 was the first to make this work at foundation-model scale. As shown in Figure 1, it involved applying attention mechanisms sequentially within rows and then within columns, capturing the complex interactions between features and the underlying relationships across different data samples.

Figure 1: Transformer architecture for TabPFN v2

Despite its success, this approach faces severe scalability limits. By performing dense attention across rows and then columns, the computational cost grows rapidly, making it prohibitively expensive as the number of samples and features increases. As a solution, TabICLv1 uses a row-compression approach, in which specialized encoders represent each row with just four tokens, making attention across samples much cheaper while maintaining competitive predictive performance. This architecture is shown in Figure 2, but it has since been adopted and extended by models like TabPFNv3, TabFM, and KumoTabular, which are among the top-ranked models in benchmarks like TabArena today.

Figure 2: Transformer architecture used in TabICLv1. For ICL, the test samples attend only to training samples, while training samples attend to each other

Pretraining with synthetic data

Apart from the architecture, another distinctive feature of TFMs is how they are built: model creators primarily use synthetic data during the pretraining phase. The determining factor in a TFM's out-of-the-box success is whether these researchers can generate synthetic data that accurately represents the distributions and relationships of real-world datasets. To build a robust foundation model, creators must simulate large-scale, diverse data that captures common enterprise quality issues like missingness, outliers, and unobserved features. While Structural Causal Models provide a standardized framework for sampling these pretraining datasets, the exact pipeline configuration depends heavily on the specific TFM.

Recent models at a glance

As the TFM landscape rapidly evolves with frequent new releases, evaluating options for enterprise deployment requires careful scrutiny. The following table highlights leading models through the lens of two practical deployment factors: scale and licensing. 

Model

Release Date

Parameters

Weights license 

TabICLv2

02/2026

~28M

BSD 3-Clause: Commercially permissive 

TabFM

06/2026

~400M

TabFM Non-Commercial License v1.0* Not commercially permissive

TabPFNv3.5

09/2026

~220M

TABPFN-3.5 License v1.0* Not commercially permissive

Mitra v2

09/2026

~77M

Apache-2.0: Commercially permissive

KumoTabular

09/2026

~215M

OpenMDW-1.1: Commercially permissive

*Commercially permissive upon agreement with the model creators.

For an exhaustive list of the current state of the art, check benchmarks like TabArena or TALENT. Given the unique trade-offs of each TFM, we recommend evaluating these models directly against your organization's data, as proprietary enterprise workloads often differ drastically from synthetic data or standard benchmarks. 

Beyond just measuring predictive accuracy, teams must also evaluate the operational cost. While TFMs save massive amounts of engineering time by eliminating bespoke training pipelines, running a foundation model on GPUs inherently costs more per prediction than executing a traditional XGBoost model on a CPU. For any given use case, evaluating a TFM means weighing that accelerated time-to-market and performance lift against the ongoing cost of GPU inference.

The GPU challenge

Inference with TFMs is optimized on GPUs. The models themselves are small enough to fit on a single GPU, but with in-context learning, the entire training set is passed to the model with every prediction. As a result, GPU memory usage grows with the number of rows and features, and can quickly be exhausted at the dataset sizes typical of enterprise ML. The need for a performant GPU leads to three problems:

  1. Provisioning GPUs: On-demand GPUs often come with long and unpredictable waiting times. Reserving a certain capacity 24/7 entails high TCO from idle capacity and sizing exercises.
  2. Managing GPUs: Getting the most out of GPUs requires good observability, especially with TFMs, where many factors affect memory usage. Engineers either spend hours building these tools and managing infrastructure or end up with heavily underutilized GPUs.
  3. Creating an external dependency: In many systems, GPUs are provisioned in separate tools, so data has to be copied out of your data platform and the inferences copied back in. This gets worse when you want to fine-tune a TFM, because governance and lifecycle become harder to manage the fine-tuned weights. 

Databricks AI Runtime for TFMs

Databricks unifies data and AI in a single open platform. AI Runtime, a fully managed serverless GPU infrastructure, offers this unified experience for enterprise GPU workloads. Experimentation and productionization have never been easier using AI Runtime, which includes features such as:

  1. Flexible, on-demand access to GPUs, including H100s that can support TFMs like TabPFN-3 at scales up to 1M rows × 200 features. 
  2. A deep learning stack with built-in observability, GPU utilization, and MLflow experiment tracking
    1. Native support for notebooks, Lakeflow Jobs, and Unity Catalog simplifies the path to production
    2. Native Docker support and remote SSH access via the Databricks CLI further improve the developer experience
  3. An in-notebook Genie Code agent with deep learning and GPU expertise capable of writing and debugging inference and training code, fixing performance bottlenecks and resolving convergence or framework errors, as shown in Video 1.
  4. Seamless deployment via Databricks Model Serving, allowing you to deploy TFMs as auto-scaling, real-time REST endpoints. For latency-sensitive use cases like credit scoring or product recommendations, this ensures your GPU-backed models can deliver low-latency predictions directly to production applications.
Video 1: Genie Code troubleshooting OOM issues directly in your notebook

With Databricks AI Runtime, you can run your TFM workloads right where your data lives, using Unity Catalog to govern and manage the lifecycle of your GPU workflows without creating any silos.

Getting started

Clone this repository to see how to run zero-shot inference with various TFMs using Databricks AI Runtime, alongside complete fine-tuning examples for classification and regression with KumoTabular. These examples showcase how AI Runtime helps you easily manage the lifecycle and integration of advanced TFM fine-tuning workflows.

Video 2 walks through the fine-tuning notebook with an H100, showing how GPU usage observability is embedded right into the notebook as well as the MLflow experiment used for metric tracking. Additionally, the workflow covers centralized governance and registration through Unity Catalog, facilitating robust version control and structured model lifecycle management. 

Video 2: Fine-tuning a KumoTabular classification model with Databricks AI Runtime. The GIF demonstrates GPU usage observability as well as model registration, governance, and version management for fine-tuned models in Unity Catalog.

Tabular foundation models promise to reduce the time to market of ML initiatives from months to days, and with Databricks AI Runtime, GPU workloads are no longer an obstacle. Your experiments can easily transition to governed, production-ready solutions, whether as high-throughput batch pipelines or real-time inference endpoints, on the same platform where your data resides, closing the gap between exploratory research and enterprise adoption. 

The machine learning landscape has reached a tipping point, and the barriers to deploying high-impact models have never been lower. Enterprises can immediately capitalize on this shift by exploring TFMs for their workloads. Don’t take our word for it; evaluate TFMs for your data to unlock performance, streamlined operations, and accurate, actionable predictions today.

FAQ

What are Tabular Foundation Models (TFMs)? 

TFMs are pretrained transformer models designed specifically for tabular data. Because they are pretrained on millions of diverse synthetic datasets, they can infer relationships in new data without needing to be trained from scratch, drastically reducing the time required to deploy machine learning models.

How do TFMs handle different types of data in a table? 

TFMs use specialized tokenizers to map mixed data types—such as numerical, categorical, and boolean features—into a single, shared latent vector space. This allows the transformer's attention mechanism to process the complex interactions between different feature types simultaneously.

Why are GPUs recommended for Tabular Foundation Models? 

TFMs are built on the transformer architecture, which relies on dense matrix multiplications that run vastly more efficiently on GPUs than CPUs. While you technically can run a TFM on a CPU, it is impractically slow for enterprise-scale data; furthermore, the massive intermediate calculations (activations) generated during in-context learning require a GPU's parallel processing power and memory bandwidth to execute in a reasonable time.

Can I fine-tune a Tabular Foundation Model? 

Yes. While TFMs provide strong predictive accuracy out of the box using zero-shot inference, you can fine-tune their pretrained weights on your specific proprietary dataset to push performance even higher for critical use cases.

How does Databricks AI Runtime help with TFMs? 

Databricks AI Runtime provides fully managed, serverless GPU infrastructure directly where your data lives. It eliminates the need to move data into external GPU tools, allowing you to run inference, fine-tune models, and manage governance through Unity Catalog without creating data silos.

Related tags:

Data Science and ML

Get the latest posts in your inbox

Subscribe to our blog and get the latest posts delivered to your inbox.

Sign up

View all blogs slice-start id="_gatsby-scripts-1" slice-end id="_gatsby-scripts-1"