IndicPrayog-tokenizer-64k

A 64,000-vocabulary Unigram (SentencePiece-style) tokenizer for Hindi, Marathi, and English, selected out of a 6-way ablation and validated with a proxy-LM training run (bits-per-byte on held-out text) — not just intrinsic fertility metrics.

• Algorithm: Unigram (SentencePiece), Metaspace pre-tokenization
• Vocab size: 64,000
• Normalization: NFC (canonical Devanagari matra composition)
• Special tokens: <pad> (0), <eos> (1), <bos> (2), <unk> (3)
• Training mix: 40% Hindi / 35% English / 25% Marathi (mix_A)
• Selected by: proxy-LM bits-per-byte (BPB) on held-out hi/mr/en text — the winner disagreement rule between intrinsic ranking and BPB never fired (same variant won both).

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("prashantcp8/IndicPrayog-tokenizer-64k")
tok.encode("यह एक परीक्षण वाक्य है।")
# [84, 39, 6019, 7010, 26]
# -> ['▁यह', '▁एक', '▁परीक्षण', '▁वाक्य', '▁है।']  (5 tokens for a 5-word sentence, fertility 1.0)

vs. Sarvam-1 (the Indic gold standard we benchmark against, 68,096 vocab — 4K larger than ours) on the exact same sentence:

from transformers import AutoTokenizer
sarvam_tok = AutoTokenizer.from_pretrained("sarvamai/sarvam-1")
sarvam_tok.encode("यह एक परीक्षण वाक्य है।", add_special_tokens=False)
# [4999, 4551, 19566, 64104, 4432, 67546]
# -> ['▁यह', '▁एक', '▁परीक्षण', '▁वाक्य', '▁है', '।']  (6 tokens — splits "है।" into 2 pieces)

Same sentence: 5 tokens (ours) vs 6 tokens (Sarvam-1) — ours keeps the sentence-final है। (word + punctuation) as one piece, Sarvam-1 splits it. This is a single illustrative example, not the benchmark; see the full FLORES-200 comparison table below.

Note: intended final home is AxisQuant/IndicPrayog-tokenizer-64k, but the cached HF token used for this session is scoped to the gemma-challenge org and the personal prashantcp8 namespace only — it lacks repo.write on AxisQuant. Published here (private) for now; transfer to AxisQuant once org write access is granted.

---

Methodology

We trained 6 tokenizer variants (A–F) on an identical 1.5 GB byte-budgeted corpus (hi 600 MB / en 525 MB / mr 375 MB, streamed from Sangraha-verified / FineWeb-Edu), varying algorithm, language mix, and pre-tokenization — to isolate which design choices actually help, rather than shipping a single default run.

ID	Algorithm	Language mix (hi/en/mr)	Pre-tokenization	Hypothesis tested
A	BPE	40/35/25	ByteLevel + digit-split	Balanced baseline
B	BPE	33/33/34	ByteLevel + digit-split	Does equal balance help Marathi enough to justify Hindi's loss?
C	BPE	45/35/20 (=raw training mix)	ByteLevel + digit-split	Control: cost of naive (unbalanced) mixing
D	Unigram	40/35/25	SentencePiece Metaspace	Literature says Unigram beats BPE on morphologically rich languages — winner
E	BPE	40/35/25	ByteLevel + UnicodeScripts (script-boundary split)	Does script-aware splitting reduce conjunct fragmentation?
F	BPE	40/35/25	ByteLevel, no digit-split	Control: cost/benefit of digit-splitting

Evaluation

Primary: FLORES-200 devtest (1,012 parallel sentences, hin_Deva/mar_Deva/eng_Latn) — parallel text makes cross-language comparison apples-to-apples (Petrov et al. 2023 methodology). Secondary: 2,000 held-out docs/language from Sangraha-verified / FineWeb-Edu.

Six metrics: fertility, compression (bytes/token), parity (Petrov-style fairness score), round-trip fidelity, Devanagari integrity (% of aksharas/conjuncts kept whole), vocab utilization.

Intrinsic results (6 ours + 8 public baselines)

Tokenizer	Group	Vocab	FLORES fert hi/mr/en	Parity hi/en	Parity mr/en	Round-trip	Dev.Integrity	VocabUtil
ours_A	ours	64,000	1.24/1.54/1.24	1.17	1.08	100.0%	93.2%	85.7%
ours_B	ours	64,000	1.26/1.51/1.25	1.18	1.05	100.0%	92.7%	86.5%
ours_C	ours	64,000	1.24/1.56/1.24	1.17	1.09	100.0%	93.3%	85.3%
ours_D (winner)	ours	64,000	1.27/1.54/1.23	1.21	1.09	100.0%	96.0%	75.2%
ours_E	ours	64,000	1.38/1.69/1.39	1.16	1.06	100.0%	94.4%	84.0%
ours_F	ours	64,000	1.22/1.51/1.21	1.18	1.08	100.0%	93.2%	86.2%
baseline_sarvam-1	baseline	68,096	1.40/1.77/1.43	1.14	1.07	100.0%	95.6%	35.0%
baseline_sarvam-30b	baseline	262,144	1.39/1.99/1.24	1.31	1.40	100.0%	94.7%	18.5%
baseline_gemma-2	baseline	256,000	1.96/3.20/1.23	1.86	2.26	100.0%	81.0%	15.2%
baseline_qwen2.5	baseline	151,643	4.76/6.69/1.26	4.42	4.62	100.0%	40.5%	20.8%
baseline_mistral-v0.3	baseline	32,768	5.33/7.34/1.37	4.55	4.66	100.0%	39.9%	57.0%
baseline_gpt2	baseline	50,257	7.82/11.16/1.23	7.41	7.86	100.0%	39.9%	60.5%
baseline_indicbert-v2	baseline	250,000	1.24/1.49/1.24	1.17	1.05	0.1%	87.3%	21.9%

We beat every non-Indic baseline (Gemma-2, Qwen2.5, Mistral, GPT-2) on Hindi/Marathi fertility and parity by a wide margin, and land within ~10% of Sarvam-1 (the Indic gold standard near our vocab size) while using a 4K-smaller vocabulary. Llama-3.1 was dropped (gated repo, license not token-bypassable). IndicBERTv2's 0.1% round-trip is expected BERT-tokenizer (WordPiece, lossy by design) behavior, not a defect.

Shortlist (top-3 by mean FLORES fertility, gated on 100% round-trip AND Devanagari integrity ≥ variant A): A, C, D. (B and F were rejected on the integrity gate.)

Proxy-LM validation (the real test)

Intrinsic metrics are proxies. To answer "which tokenizer makes a model learn best," each shortlisted candidate trained an identical ~28M-param Llama-style dense LM from scratch (hidden=320, 6 layers, 5 heads, RoPE, SwiGLU, tied embeddings) on an identical 300M-token slice (same source documents, tokenized per-candidate), 3000 steps, same seed, then measured bits-per-byte (BPB) on held-out hi/en/mr text — BPB normalizes by raw bytes, not token count, making it the fair cross-tokenizer comparison metric.

Variant	mean BPB	hi	en	mr
D (winner)	1.0404	0.7229	1.6664	0.7320
A	1.0671	0.7324	1.7201	0.7489
C	1.0686	0.7316	1.7226	0.7516

D wins on every language individually, not just on average — confirming the Unigram algorithm's advantage on morphologically rich Devanagari script carries through to actual downstream LM quality, matching the intrinsic-metric ranking (no override needed).

---

Bugs found and fixed during this ablation

Six real bugs were caught and fixed via direct verification (not just accepting first-pass numbers) before this tokenizer was selected:

1. add_prefix_space=True spurious "Ġ" markers. ByteLevel injected a leading-space marker at every upstream pretokenizer split boundary, not just at string start — corrupting round-trip fidelity around digits/script transitions. Fixed with add_prefix_space=False.
2. UnicodeScripts() silently dropping whitespace-only pretokens. When a lone space pretoken (left behind after Digits() split off a digit) reached UnicodeScripts() in isolation, it was dropped entirely, silently deleting a character. Fixed by running UnicodeScripts() before Digits() in the pretokenizer Sequence.
3. ByteLevel's word-boundary regex splits Devanagari at every matra (the big one). ByteLevel's internal regex (use_regex=True, default) groups only Unicode category Letter into a "word" — Devanagari matras are category Mark, not Letter — so every Devanagari syllable was pre-split at matra boundaries (भ|ा|रत instead of भारत), making it structurally impossible for BPE to ever merge across a matra. This alone caused ~3.3–5 tokens/word Hindi/Marathi fertility and 40.6% Devanagari integrity. Fixed with a custom Split regex that groups Letter+Mark runs, combined with ByteLevel(use_regex=False).
4. Space-isolation regression (introduced by the fix above). The first fix regex isolated every space as its own pretoken (unlike GPT-2's convention of attaching a leading space to the following word), so BPE could never merge a space into a word — nearly doubling English fertility (1.26→2.14). Fixed by attaching a leading space to the following word/digit run: " ?[^\s\d]+| ?\d+|\s+".
5. Eval script's non-NFC-aware round-trip check. All "ours_*" variants showed 96.9% round-trip despite direct spot-checks showing 100%. Root cause: our tokenizer applies NFC normalization by design, so decode(encode(s)) legitimately returns NFC(s), which can differ as a raw string from non-canonically-composed FLORES source text while being visually/semantically identical. Fixed the eval script to NFC-normalize both sides before comparing — the correct definition of "lossless" for an NFC-normalizing tokenizer.
6. Proxy-LM tied-embedding weight-init bug. nn.Embedding's PyTorch default init is N(0, 1); since the LM head is tied to the embedding, that same std≈1.0 matrix produced logits directly, giving step-0 loss ~307–309 nats instead of the correct ~ln(64000)≈11.07 for a freshly initialized 64K-vocab LM — wasting most of training recovering from a broken init rather than learning real signal, and producing an unreliable first BPB comparison. Fixed with GPT-2/Llama-style N(0, 0.02) init applied to all Linear/Embedding weights. Re-running with the fix confirmed step-0 loss ≈11.1 for all three candidates and changed nothing about the final ranking (D still wins), but the corrected numbers above are the ones that should be trusted.

Known limitation

tokenizers==0.22.2's UnigramTrainer(byte_fallback=True) silently ignores the byte_fallback kwarg (Ignored unknown kwargs option byte_fallback) — so <unk> is not structurally impossible for this Unigram tokenizer as it would be for the BPE variants (A/B/C/E/F use ByteLevel byte-fallback, which IS structurally lossless). In practice, round-trip fidelity measured 100.0% on the full FLORES-200 eval set, so OOV characters were not encountered in testing.

Reproducibility

Full corpus-sampling, training, eval, shortlist, and proxy-LM scripts are included in AxisQuant/IndicPrayog-tokenizer-ablations (all 6 variants + full results), together with the original experiment plan. Fixed seed (1234) throughout; corpus streamed from ai4bharat/sangraha and HuggingFaceFW/fineweb-edu; eval on haoranxu/FLORES-200.

The proxy-LM checkpoint that produced the winning BPB result above is published (private) at prashantcp8/IndicPrayog-proxy-lm-D.