dhivehi-byt5-latin2thaana-keymap-v1

ByT5 fine-tune for Latin → Dhivehi transliteration that emits an ASCII phonetic keymap instead of Thaana UTF-8 bytes. A short post-processing function maps the ASCII output back to Thaana script.

Because Thaana codepoints (U+0780–U+07BF) occupy three UTF-8 bytes and ByT5 generates one decoder step per output byte, decoding 100 Thaana characters costs 300 decoder steps. Emitting the keymap form (1 byte per character) and converting after generate() finishes gives an end-user-equivalent result at ~3× lower decoder cost.

The model warm-starts from Neobe/dhivehi-byt5-latin2thaana-v1; only the decoder retargets to the new output alphabet.

Usage

The model's raw output is ASCII (Segha phonetic keyboard layout). You need the keymap_to_thaana conversion function from the project repo to render Thaana.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from keymap import keymap_to_thaana  # from div-transliteration repo

MODEL_ID = "str33t/dhivehi-byt5-latin2thaana-keymap-v1"
tok = AutoTokenizer.from_pretrained(MODEL_ID)
mdl = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID)

text = "aharumen anekkaa ves bahdhalu vee"
inputs = tok(text, return_tensors="pt")
out = mdl.generate(**inputs, num_beams=4, max_new_tokens=512)
ascii_keymap = tok.decode(out[0], skip_special_tokens=True)

thaana = keymap_to_thaana(ascii_keymap)
print(thaana)

Keymap

The Segha phonetic layout maps every Thaana letter / fili / sukun to a unique ASCII byte (see keymap.py in the project repo for the exact tables, taken verbatim from jawish/jtk). Conversion is deterministic and lossless on in-layout characters; digits, punctuation, and whitespace pass through.

Training

• Base model: Neobe/dhivehi-byt5-latin2thaana-v1
• Corpus: alakxender/dhivehi-transliteration-pairs (Thaana side converted to keymap via keymap.py)
• Length-diverse augmentation to mitigate the short-input hallucination and long-input degradation modes documented in the project's MODEL_NOTES.md.

Full methodology, ablations, and results: project README.

License

Apache 2.0 (inherited from the base model and ByT5).