We ran the same document through OpenAI's tokenizer in English, Malay and Simplified Chinese, using the UN's own translations. The Malay version used 56% more tokens than the English one. On OpenAI's API, which bills by the token, that means that at the same model and price per token the same content, in the UN's own Malay translation, cost more to process than in English.
Contents
Quick facts
- The Malay translation needed 1.56 times the English version's tokens on o200k_base, the tokenizer OpenAI's open-source tiktoken library assigns to GPT-4o through GPT-5.
- The Simplified Chinese translation needed 1.14 times.
- On cl100k_base, the older tokenizer used by GPT-4 and GPT-3.5, the Malay text needed 1.95 times.
- The test text: the preamble and 30 articles of the Universal Declaration of Human Rights, in translations checked against the UN human rights office's own copies.
What we measured
AI models do not read words. They read tokens, small chunks of text, and OpenAI's API price list is quoted per million of them. Text that splits into more tokens costs more to send and fills more of the model's context window.
Rest of World reported on 26 October that researchers call this a "tokenizer tax". Its report did not discuss Malay or Chinese, so we measured them ourselves.
We used tiktoken, the tokenizer library OpenAI publishes, on the Universal Declaration of Human Rights. Our digital copies came from the NLTK research collection and we checked each one against the UN human rights office's own PDF translation. A text-similarity comparison scored them between 98.4% and 99.7% alike, with the differences down to page headers and numbering. We counted from the preamble to Article 30 and left out titles and introductions.
What it means in ringgit
English took 1,971 tokens. Malay took 3,069. Simplified Chinese took 2,244. At the same price per token, every RM1 spent processing the English version would become about RM1.56 for the Malay translation and RM1.14 for the Chinese one.
Malay uses the same Latin alphabet as English, so script alone seems an unlikely explanation. Our reading is that Malay's longer words, such as "pengiktirafan" and "kebebasan", get split into more pieces. That is our interpretation, not something this test measured.
Old tokenizer against new
On this document, o200k_base produced smaller gaps than cl100k_base, the tokenizer behind the older GPT-4 and GPT-3.5. Malay fell from 1.95 to 1.56 times English, and Chinese from 1.67 to 1.14 times.
What we did not test
This covers two OpenAI tokenizers only, and OpenAI's tiktoken library does not say which tokenizer its newest models use. Google's Gemini, which, as we have reported, sells AI plans in Malaysia from RM23.99 and offers a one-million-token model here, and Anthropic's Claude were not tested, so their figures may differ. We left Tamil out: the copy in the research collection is a different translation from the UN office's current one, and the UN's Tamil PDF does not extract as text, so we could not measure the official wording. We measured text going in, not a model's answers, response speed, system prompts or a full monthly bill, and a ChatGPT subscriber on a flat monthly plan does not pay per token at all. One formal document is also one sample: everyday chat Malay, full of English loanwords and short forms, may split differently.
