Anthropic's extravagant tokenizer complicates AI pricing

Claude looks substantially more token-hungry than OpenAI's GPT-5.x, thanks to the new tokenizer that Anthropic shipped with recent releases. Large language models (LLMs) use tokenizers to handle the mapping of text into tokens. There's no set definition of a token, but they're typically a set of three or four characters that are mapped to the integers LLMs actually process.