研究显示部分语言使用 AI 模型所需 token 最多高出 17 倍

A new study from the Coalition for Language Equity in AI Research shows that AI models require significantly more tokens to process certain languages than others. These differences mean that users of underrepresented languages risk paying more for the same services, reports Rest of World.

An LLM’s tokenizer breaks text down into blocks, or tokens, but it does so much more optimally for languages with abundant digitized data, the researchers explain. They found that among the 20 languages with the largest web presence, less common languages used from 1.5 to 5 times as many tokens as the same prompt in English. When the study was expanded to 200 languages, the difference increased to as much as 17 times in one of the models examined.

Since most LLM providers charge by tokens used, that translates to much higher costs. In addition, more tokens mean longer response times and a smaller amount of information that’s able to fit into the AI models’ context windows.

The researchers describe this as a hidden “tokenizer tax” and warn that it exacerbates digital inequality.

This disparity is not limited to languages with few speakers, the researchers note. “If we sort by speakers instead of web presence, the tax lands on enormous populations: Punjabi (~200 million speakers), Bengali (~266 million), Urdu (~288 million), Burmese (~36 million), Amharic (~36 million). This means hundreds of millions of people billed at multiples of the English rate for the same intelligence,” the study says.

Languages that use writing systems not as commonly found in the AI models’ training data are particularly affected. Examples include Burmese, Tibetan, Khmer, and Lao. Languages that use the Latin alphabet, on the other hand, often have an advantage, even when spoken by relatively few people.

Related: CIOs grapple with subpar global genAI models