# The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India

**Authors:** Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin Sam  
**Published:** 9 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.13058) · [PDF](https://arxiv.org/pdf/2611.13058) · [Code](https://github.com/HavonAI/indic-token-tax) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.13058.html)

## Key numbers

- **35%**: no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries
- **25**: on all 25 languages of India available in FLORES-200
- **13,732**: and on 13,732 human-translated Hinglish-English social media pairs from PHINC

## Abstract

Commercial language model APIs bill per token, so a tokenizer that needs more tokens for the same content in one language makes that language more expensive, slower, and able to fit less text in a fixed context window. We measure this disparity for nine public tokenizers, including those of GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo and four Sarvam models, on all 25 languages of India available in FLORES-200 (2,009 professionally translated sentences aligned with English) and on 13,732 human-translated Hinglish-English social media pairs from PHINC. On GPT-4o's tokenizer the same content needs between 1.57 times (Hindi) and 5.01 times (Odia) the tokens of English, and 13.7 times for Santali in Ol Chiki script; GPT-4's tokenizer needs up to 12.8 times. Sarvam-30B/105B is the only third-party tokenizer that stays below 2.6 times on every language (median 1.53). In an illustrative scenario using official list prices accessed on 5 November 2026 and stated assumptions, a support bot handling 100,000 chats a month on GPT-4o would cost about INR 56,640 in English and INR 102,829 in Marathi; these are estimates, not measured production costs. Against an openly licensed Hindi and Marathi morphological segmentation set, no tokenizer places more than 35% of its word-internal boundaries on morpheme boundaries (boundary F1 at most 0.48). In a companion paper we present a tokenizer that reduces this tax. Code, pinned artifacts and raw token counts are available at https://github.com/HavonAI/indic-token-tax.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.