How VocabX cuts real-world inference costs by 77% with specialized SLM vocabularies
VocabX selects tokens by cost rather than frequency and simultaneously rebuilds subword segmentation and multi-word merging, compressing the vocabulary of domain-specific small models to a smaller scale and reducing inference costs by 77% in real-world applications. Taking a scoring model as an example, generating one token costs 7.4 times as much as reading one token, and the output-layer vocabulary shrinks from about 152,000 rows to about 65,000 rows. This method targets the memory bandwidth bottleneck of small models such as Qwen3-4B at medium batch sizes, whereas methods such as FVT, MWT, and in-place BPE select tokens only by frequency and address only a single level.
VocabX selects tokens by cost rather than frequency and simultaneously rebuilds subword segmentation and multi-word merging, compressing the vocabulary of domain-specific small models to a smaller scale and reducing inference costs by 77% in real-world applications. Taking a scoring model as an example, generating one token costs 7.4 times as much as reading one token, and the output-layer vocabulary shrinks from about 152,000 rows to about 65,000 rows. This method targets the memory bandwidth b
Matched fieldsBody
