Understand this story
A new study by the Coalition for Language Equity in AI Research finds that AI models require significantly different numbers of tokens to process different languages. Among the 20 languages with the highest web presence, less common languages use 1.5 to 5 times as many tokens as English to process the same prompt; after the study expanded to 200 languages, the gap in one of the models examined reached as high as 17 times. Because most LLM providers bill by token, this drives up usage costs and also brings longer response times and smaller available context windows, which researchers call a hidden tokenizer tax.
AI-generated from public reports
Latest development · Report
Study shows some languages require up to 17 times more tokens to use AI modelsDevelopments
1 confirmed developments
Study shows AI models need more tokens to process some languages
Source:Computerworld
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Computerworld
Study shows some languages require up to 17 times more tokens to use AI models
A new study by the AI Language Equity Research Consortium found significant differences in the number of tokens AI models need to process different languages: among the 20 languages with the highest web presence, less common languages use 1.5 to 5 times as many tokens as English to process the same prompt. After expanding the study to 200 languages, the gap reached as high as 17 times in one of the models examined. Because most LLM providers charge by token, this drives up usage costs and also brings longer response times and smaller available context windows, which researchers call a hidden tokenizer tax.