adityaghai07/ag_hindi_bpe_tokenizer_32k

Hindi BPE Tokenizer (32K)

Character-level BPE tokenizer for Hindi (Devanagari script) with Metaspace pre-tokenization. Trained on Hindi Wikipedia.

• Vocab size: 32,000
• Algorithm: BPE with Metaspace
• Training data: Hindi Wikipedia (~519 MB, 688K lines)
• Normalization: Unicode NFC
• Special tokens: <|pad|> <|bos|> <|eos|> <|unk|> <|sep|> <|mask|> <|user|> <|assistant|>

Fertility ~1.5-1.8 tokens/word vs ~6-8 for GPT-2 on Hindi text. Preserves Devanagari conjuncts, matras, and combining marks. Handles code-mixed Hindi-English.

Usage

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("adityaghai07/ag_hindi_bpe_tokenizer_32k")

ids = tokenizer.encode("भारत एक महान देश है।")
print(tokenizer.decode(ids))

Source

GitHub