It costs a lot more to use AI models in some languages

A new study from the Coalition for Language Equity in AI Research shows that AI models require significantly more tokens to process certain languages than others. These differences mean that users of underrepresented languages risk paying more for the same services, reports Rest of World . An LLM’s tokenizer breaks text down into blocks, or tokens , but it does so much more optimally for languages with abundant digitized data, the researchers explain.