Study Says Non-English Prompts Have Higher Token Costs
English text is being prepared.
A study published last month by the Coalition for Language Equity in AI Research found that AI models require far more tokens to process prompts in some languages than in English, which researchers call a hidden token tax. Among the 20 most popular languages, the most expensive language uses 1.5-5 times as many tokens as the cheapest language; when expanded to 200 languages, the token cost gap for one model increases by as much as 17 times. The study points out that a higher number of tokens means higher latency and lower effective context capacity, with the impact most severe in regions with limited computing and connectivity.
Why read this
AI score 56
The study quantified the token cost gap between different languages, which readers can use to understand the cost and latency differences faced by users of minority languages.
Topics
Body
Coalition for Language Equity in AI Research 上个月发表的一项研究发现，AI 模型处理部分语言提示所需的 token 量远超英语。研究人员称之为隐性 token 税，意味着小语种用户的 AI 成本更高。研究发现，即使互联网最流行的 20 种语言，token 使用量也存在差异。最昂贵语言使用的 token 数量是最便宜语言的 1.5-5 倍。当研究人员将范围扩大到 200 种语言，差异进一步扩大，一个 AI 模型的 token 成本差距甚至增加了 17 倍。研究指出：位于班加罗尔的初创公司可能需要为每次提示和完成支付附加费，而其位于奥斯汀的竞争对手则无需支付。token 税的影响远不止成本。token 数量越多，延迟就越高，有效上下文容量就越低。在计算和连接本就有限的地区，这些惩罚最为严重。token 不公平限制了 AI 经济的参与范围。