Study Says Non-English Prompts Have Higher Token Costs
A study published last month by the Coalition for Language Equity in AI Research found that AI models require far more tokens to process prompts in some languages than in English, which researchers call a hidden token tax. Among the 20 most popular languages, the most expensive language uses 1.5-5 times as many tokens as the cheapest language; when expanded to 200 languages, the token cost gap for one model increases by as much as 17 times. The study points out that a higher number of tokens means higher latency and lower effective context capacity, with the impact most severe in regions with limited computing and connectivity.
Why read this
AI score 56
The study quantified the token cost gap between different languages, which readers can use to understand the cost and latency differences faced by users of minority languages.
Topics
Body · AI translation
A study published last month by the Coalition for Language Equity in AI Research found that AI models require far more tokens to process prompts in some languages than in English. Researchers call this the hidden token tax, meaning that users of minority languages pay more for AI. The study found that even among the 20 most popular languages on the internet, token usage varies. The most expensive language uses 1.5 to 5 times as many tokens as the cheapest language. When the researchers expanded the scope to 200 languages, the disparity widened further, with the token cost gap for one AI model increasing by as much as 17 times. The study points out that a startup in Bangalore may need to pay a surcharge for every prompt and completion, while its competitor in Austin does not. The impact of the token tax goes far beyond cost. The more tokens, the higher the latency and the lower the effective context capacity. These penalties are most severe in regions where computing and connectivity are already limited. Token inequity restricts the scope of participation in the AI economy.