Understand this story
On October 28, 2026, Cartesia published the second article in a series, using two types of scaling laws—compute-to-loss and loss-to-loss—to compare byte-level Transformers, BPE, BPE-NP with pre-tokenization removed, BLT-like, and H-Net. Cartesia states that BPE tokenization is nearly 5 times more compute-efficient than byte-level models in pretraining, but at the same validation bpb, OLMES downstream loss is clearly worse, equivalent to a compute gap of about 1.9 times; H-Net simultaneously obtains the compute efficiency brought by compression and the transfer capability of byte-level models, with a direct fit of compute-to-OLMES at about 2.06 times. The article also points out that the manual regex step of pre-tokenization has a clear impact on both metrics, and discusses the prompt boundary problem of BPE.
AI-generated from public reports
Latest development · Report
Cartesia Research: From Tokens to Bytes, How Tokenization Priors Affect Language Model Downstream TransferDevelopments
1 confirmed developments
Cartesia study compares scaling laws of byte-level and tokenized language models
Source:English text is being prepared.
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Cartesia Products and ResearchFirst-party
Cartesia Research: From Tokens to Bytes, How Tokenization Priors Affect Language Model Downstream Transfer
Cartesia has released the second article in its series, using two types of scaling laws, compute-to-loss and loss-to-loss, to compare byte-level Transformers, BPE, BPE-NP with pre-tokenization removed, BLT-like, and H-Net. The conclusion is that BPE tokenization is nearly 5 times more compute-efficient than byte-level models in pretraining, but at the same validation bpb, OLMES downstream loss is significantly worse, equivalent to a compute gap of about 1.9 times; H-Net simultaneously obtains the compute efficiency brought by compression and the transfer capability of byte-level models, with a direct compute-to-OLMES fit of about 2.06 times. The article also points out that the manual regularization step of pre-tokenization has a significant impact on both metrics, and discusses the prompt boundary issue of BPE.