Cartesia Research: From Tokens to Bytes, How Tokenization Priors Affect Language Model Downstream Transfer
Cartesia has released the second article in its series, using two types of scaling laws, compute-to-loss and loss-to-loss, to compare byte-level Transformers, BPE, BPE-NP with pre-tokenization removed, BLT-like, and H-Net. The conclusion is that BPE tokenization is nearly 5 times more compute-efficient than byte-level models in pretraining, but at the same validation bpb, OLMES downstream loss is significantly worse, equivalent to a compute gap of about 1.9 times; H-Net simultaneously obtains the compute efficiency brought by compression and the transfer capability of byte-level models, with a direct compute-to-OLMES fit of about 2.06 times. The article also points out that the manual regularization step of pre-tokenization has a significant impact on both metrics, and discusses the prompt boundary issue of BPE.
Cartesia has released the second article in its series, using two types of scaling laws, compute-to-loss and loss-to-loss, to compare byte-level Transformers, BPE, BPE-NP with pre-tokenization removed, BLT-like, and H-Net. The conclusion is that BPE tokenization is nearly 5 times more compute-efficient than byte-level models in pretraining, but at the same validation bpb, OLMES downstream loss is significantly worse, equivalent to a compute gap of about 1.9 times; H-Net simultaneously obtains th
Matched fieldsBody
