OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration
01 The other side of intelligence-per-cost
Intelligence built for the real world must answer a concrete question: how can we use more reasonable time, compute, and engineering investment to continuously improve a model's ability to solve real problems?
Pursuing intelligence-per-cost means looking not only at how fast a model runs, but also at how much time and engineering cost it takes to validate a single data strategy.
For the former, Ling recently gave an answer: a native hybrid reasoning model, Ling-3.0-flash with 124B total parameters and 5.1B activated parameters, which does not simply pile on parameter scale but pursues the intelligence that can be converted per unit of compute.
The latter is hidden in the inputs during R&D that are not easily seen: how long it takes to prepare a batch of data, how much manual operation is needed to adjust a quality standard, and whether the data sources and processing of an experiment can be clearly traced.
Letting researchers wait less and repeat fewer operations, and giving them more opportunities to validate questions, is the value that data engineering can contribute.
Around this goal, Ling built OmniTable — a unified wide-table system for petabyte-scale LLM data curation and exploration. The related paper received the VLDB 2026 Best Industry Paper Award.
02 Why does validating a single data strategy take so long?
When researchers want to validate a data strategy, they often start with a concrete question: can increasing the proportion of a certain type of high-quality sample improve the model's performance on a specific task?
But before starting subsequent training, the team needs to gather data from different sources, complete cleaning, deduplication, quality scoring, and safety checks, and then filter, combine, and export according to the experiment requirements.
The paper documents a real SFT data curation task at Ling: gathering 8 data sources and completing the production of 12 data features — some involving rule-based processing, and others requiring model calls for quality assessment.
Under the original approach, data ingestion took about 2 days, computing the 12 features took about 9.5 days, and filtering and export took about 2.5 days, for a total of about 14 days.
With OmniTable, the end-to-end processing cycle for the same task was shortened to about 2.5 days, reducing the data processing cycle by about 82.1%. The number of manual operation steps in the process also dropped from about 45 to about 12, a reduction of about 73.3% — this metric reflects the simplification of the operational process and does not equate to a proportional reduction in labor cost.
It should be specially noted thatthe reduction from 14 days to 2.5 days is the processing cycle of this specific data curation task, not the complete model training cycleWhat it brings is less data waiting and fewer repeated operations before subsequent training begins, allowing researchers to move a data strategy to a trainable, verifiable stage more quickly. Whether the strategy is effective still has to be answered by subsequent training and evaluation.
OmniTable shortened a real SFT data curation task from about 14 days to about 2.5 days
03 Where did the 14 days of waiting go?
These 14 days include not only the time for feature computation, but also the overhead of data ingestion, task orchestration, and filtering and export.
In traditional data systems, data and features from different sources and at different stages are usually scattered across a large number of physical tables. Each time a quality score or safety label is added, the team may need to create new tables, write additional scripts, and re-orchestrate tasks; when the raw data or processing rules change, the related features must also be updated layer by layer along the pipeline.
In the aforementioned task, the original process involved 35 physical tables, 24 data pipelines or scripts, and about 45 operation steps. Another real case is more intuitive: to add one feature to web data, engineers had to connect 106 tables one by one on the task orchestration canvas.
Data production thus becomes an ever-expanding "pipeline maze": more and more tables, increasingly complex dependencies. Researchers not only have to wait for computation results, but also invest effort in maintaining processes, troubleshooting, and confirming data sources — sometimes they have to watch the pipeline at night, and once it fails, they have to troubleshoot and retry.
The complexity of data engineering ultimately translates into waiting time for data strategy validation. This is an easily overlooked loss in intelligence-per-cost.
The "pipeline maze" in petabyte-scale LLM data production
04 OmniTable: leave complexity to the system, give time back to research
The core idea of OmniTable can be summed up in one sentence: logically one wide table, physically continuously adaptive.
For researchers, data from different sources and their features can be used like a continuously growing table: adding a quality score is like adding a column to the table; adjusting filtering criteria does not require rebuilding a pipeline. Researchers only need to specify which data and features are needed, and the system handles the specific computation, updates, and storage.
The changes are mainly reflected in three aspects:
- Less repeated orchestration. Feature dependencies are automatically sorted out by the system, so when adding or adjusting a feature, there is no need to build pipelines table by table; tasks requiring model scoring are automatically scheduled to GPU resources, and the system automatically isolates and records data that fails processing, reducing the chance that an individual data anomaly causes an entire batch to fail.
- Accumulated expert experience. Computation parameters no longer rely on repeated trial and error based on personal experience — the system automatically configures them according to data scale and historical tasks, raising the first-submission success rate for new team members from about 60% to over 90%.
- Traceable processes. Where data comes from, what processing it went through, and which version it formed can all be recorded and queried, so the sources and processing of experimental data are no longer an "unclear account."
OmniTable does not replace model training, nor does it directly judge whether a data strategy is effective. What it solves is: enabling a data strategy to enter subsequent training and evaluation with less waiting and fewer repeated operations.
OmniTable system architecture: the upper layer provides a unified wide-table view, while the lower layer retains a continuously adjustable storage and compute system
05 Running continuously on 35 PB of data
Efficiency gains are truly meaningful only when they hold over the long term in large-scale production environments.
Currently, the data managed by OmniTable exceeds 35 PB, covering more than 305 billion records and spanning multiple types such as web pages, code, PDFs, and SFT data.
These data do not stay unchanged after a single processing pass. As research goals and quality standards shift, the team needs to continuously add new data sources and features, recompute some results, and combine data according to new experimental requirements. The larger the data scale and the more frequent the changes, the more complex the engineering chain affected by a single strategy adjustment—every one-point improvement in engineering efficiency multiplies the savings in R&D investment.
The real test is this: as scale keeps growing, can efficiency avoid collapsing? Two sets of tests in the paper provide the answer—on the web wide table, as the amount of data covered by a single export task grew from 1 TB to 25 PB, the system's export throughput remained essentially stable at 18–23 TB/hour; as the wide table's logical columns grew from 200 to 2,500, exceeding the underlying storage engine's physical limit of about 1,200 columns, query latency rose by only about half. Behind this is the system's continuous background data organization and layout optimization; if it were turned off, throughput at the same scale would drop severalfold.
Data volume grew by about four orders of magnitude and feature columns broke through the storage engine's physical limit, yet OmniTable's export throughput and query latency remained essentially stable
OmniTable runs continuously in such production environments, supporting Bailing's daily data governance and exploration.
06 How can data engineering improve the intelligence-efficiency ratio?
In Ling-3.0-flash the intelligence-efficiency ratio has an intuitive measure: the level of intelligence that can be provided per unit of compute cost—using 5.1B activated parameters to deliver capabilities beyond flagship models of a comparable parameter scale.
In the model R&D process, the intelligence-efficiency ratio has another side: how much time and engineering resources must be invested from posing a question to obtaining a verifiable result.
A shorter data production cycle does not mean model performance will necessarily improve. But it can enable research teams to:
- judge more quickly whether a data choice is worth pursuing;
- compare different quality standards and data mixes more conveniently;
- trace the source and processing of experimental data more clearly;
- reduce repeated investment in building tables, orchestrating tasks, and maintaining scripts.
Under the same time and engineering conditions, researchers can therefore gain more opportunities to validate questions, identify ineffective directions earlier, and let effective data strategies enter subsequent training and evaluation faster.
This is precisely data engineering's contribution to the intelligence-efficiency ratio:not drawing conclusions for the model, but reducing the waiting and engineering cost required to obtain reliable conclusions.
07 Bringing every data exploration closer to the answer
VLDB is a major international conference in the field of data management, now in its 52nd edition, and its industrial track focuses on systems and practices that can solve large-scale data problems in real production environments.
OmniTable receiving the VLDB 2026 Best Industrial Paper Award is recognition of this large-scale data engineering practice. For Bailing, what matters more is the practical problems it solves: how to let researchers reduce waiting and repetitive operations, how to make data processing easier to trace and reproduce, and how to provide stable support for curated, processed, and validated data to enter subsequent training.
From Ling-3.0-flash to OmniTable, Bailing continues to improve the intelligence-efficiency ratio along two dimensions—model operation and R&D engineering: raising the model capability that each unit of compute can support, and improving the effective validation efficiency that each unit of engineering investment can support.
Intelligence for the real world requires models, data, and systems to work together. Bailing will also continue to start from these concrete engineering problems and explore a higher intelligence-efficiency ratio among model capability, operational efficiency, and R&D investment.
The related paper is now public:https://arxiv.org/abs/2609.11148v1
《OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration》
Click "Read original article" at the end of this post to view the full paper.
Technical sharing dataPrevious post: The Ming-Image-0.1-Design series is open-sourced, entering real design workflowsNext post: Ling-3.0-flash-VL open-sourced: letting visual capabilities serve real tasks