New Study Says AI Model Token Consumption Varies by Up to 17 Times Depending on Language

New Study Says AI Model Usage Costs Are Affected by Different Languages, with Token Consumption Differing by Up to 17 Times

2026-10-30 12:25 IT Home - Qingyuan

Thanks to IT Home user Coje_He for the tip submission!

IT Home, October 30 news: Users of less widely spoken languages may need to pay higher fees than users of English and other mainstream languages when using AI services.

On the 28th local time, according to foreign media outlet Rest of World, a study published last month by the AI Language Equity Research Consortium found that when AI models process different languages, the number of Tokens consumed may differ vastly.

A Token is the basic unit by which AI models process data such as text, created when a tokenizer splits text, images, or audio. Researchers call the additional costs thus generated for different languages a hidden "tokenizer tax," and languages with less web content are especially vulnerable.

Even among the 20 most commonly used languages on the internet, the study found significant differences in Token consumption, with the highest being 1.5 to 5 times the lowest. If the testing scope is expanded to 200 languages, the gap widens further. In one of the AI models, Token consumption across different languages differed by up to 17 times.

The research report points out that a startup in Bangalore may pay a higher cost each time it inputs instructions to an AI and gets a response than its competitor in Austin. The problems brought by the "tokenizer tax" are not limited to rising costs. The more Tokens consumed, the slower the AI response, and the less context it can effectively process.

Pratyush Choudhury, co-founder of Activate AI, India's first venture capital fund focused on AI investment, pointed out that when using non-English languages, the same context capacity can hold fewer documents and conversation records, and AI generates answers with more difficulty.

Choudhury explained using teaching as an example: "For a teacher to teach students well, they have to remember what mistakes students have made before and what explanations they have received. If, when using a certain language, these historical records quickly fill up the context window, ultimately even the quality of teaching will be affected."

Choudhury believes that language processing efficiency should become a basic requirement for AI products, just like accuracy, speed, and reliability. "Requiring completely equal treatment for all languages may be unrealistic, but the long-standing huge gap still needs a reasonable explanation and an improvement plan. I hope all major AI labs will publish evaluation results for each language, explain how much it costs to complete tasks of the same difficulty in different languages, and commit to improving the most weakly supported languages with quantifiable progress."

IT Home learned from the report that one way to improve this problem is to improve the quality of the underlying data.

Raoul Nanavati, co-founder and CEO of Mumbai-based AI voice infrastructure startup Navana.ai, believes: "Open data projects are important in this regard. For example, our RESPIN project, carried out in cooperation with the Indian Institute of Science and funded by the Gates Foundation, collected text and read-aloud recordings provided by native speakers of 38 dialects."

However, the AI Language Equity Research Consortium found that what determines Token consumption is not only the number of speakers of a language; the writing system used is equally important.

For example, Asturian has few speakers and is not included in the Common Crawl dataset, but the Latin alphabet gives it an advantage. Languages in the same Romance family, such as Italian, are covered in the data, which also benefits Asturian.

Dimante Reddy, CEO of Maya Research, pointed out that many existing tokenizers are still mainly optimized for languages with abundant data resources, such as English, and the fundamental problem lies in the lack of high-quality digitized data.

Reddy emphasized that improving the training data itself is not enough: "Token prices are falling quickly, and the direct impact of cost differences should gradually diminish. Better multilingual data and synthetic data will also help improve model quality. But to truly narrow the tokenization gap between languages, AI labs still need to specifically improve the design of next-generation tokenizers so that they can better handle these languages and writing systems."