babylm_telugu_2026

A GPT-2-style causal language model trained on Telugu, built for the BabyLM 2026 shared task. This repo contains 6 model checkpoints comparing two training strategies — random data ordering vs. curriculum learning — each run with 3 different random seeds, enabling statistically robust comparison.

---

Repo Structure

babylm_telugu_2026/
├── random_seed1/        # Random data order, seed 1
├── random_seed2/        # Random data order, seed 2
├── random_seed3/        # Random data order, seed 3
├── curriculum_seed1/    # Curriculum learning, seed 1
├── curriculum_seed2/    # Curriculum learning, seed 2
└── curriculum_seed3/    # Curriculum learning, seed 3

Each subfolder is a self-contained GPT-2 checkpoint (Safetensors format) loadable independently via from_pretrained.

---

Experimental Design

The core research question is: does curriculum learning improve sample-efficient pretraining for Telugu?

Condition	Data Order	Seeds
random_seed*	Shuffled uniformly at random	1, 2, 3
curriculum_seed*	Simple → complex, ordered by sentence length (bytes per line)	1, 2, 3

The curriculum strategy orders training examples from shortest to longest sentence (by byte count), so the model encounters simpler linguistic structures before more complex ones. This mirrors cognitively plausible language acquisition and follows the byte-ranked curriculum approach explored in prior BabyLM submissions.

Running 3 seeds per condition allows variance estimation and guards against seed-specific artefacts — a known issue in low-data training regimes.

---

Model Details

Property	Value
Architecture	GPT-2 (decoder-only Transformer)
Language	Telugu (te)
License	CC-BY-4.0
Training framework	Hugging Face Transformers
Model format	Safetensors

---

Training Data

Trained on pulipakav-1/translated-babylm-telugu — a Telugu translation of the English BabyLM 2026 strict corpus produced using IndicTrans2.

Split	Rows
Train	11.6M
Validation	1.2M
Test	1.1M

• Source corpus: BabyLM-community/BabyLM-2026-Strict
• Translation model: ai4bharat/indictrans2-en-indic-1B
• License: CC-BY-4.0

---

How to Use

Load any individual checkpoint by pointing from_pretrained at the subfolder:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Choose any of the 6 checkpoints
checkpoint = "pulipakav-1/babylm_telugu_2026/curriculum_seed1"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)

prompt = "తెలుగు భాష చాలా అందమైనది"  # "Telugu language is very beautiful"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Comparing all runs

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo = "pulipakav-1/babylm_telugu_2026"
checkpoints = [
    "random_seed1", "random_seed2", "random_seed3",
    "curriculum_seed1", "curriculum_seed2", "curriculum_seed3",
]

text = "తెలుగు భాషలో సాహిత్యం చాలా సమృద్ధంగా ఉంది."

for ckpt in checkpoints:
    path = f"{repo}/{ckpt}"
    tokenizer = AutoTokenizer.from_pretrained(path)
    model = AutoModelForCausalLM.from_pretrained(path)
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        loss = model(**inputs, labels=inputs["input_ids"]).loss
    print(f"{ckpt:25s}  perplexity: {torch.exp(loss).item():.2f}")

---

Evaluation

⚠️ Results will be updated as evaluations are completed.

Planned metrics, reported as mean ± std across the 3 seeds per condition:

Metric	random (mean ± std)	curriculum (mean ± std)
Test perplexity	—	—
BLiMP-style accuracy	—	—

---

Intended Use

• Research into curriculum learning for low-resource, morphologically rich languages
• Baseline comparisons for Telugu language modeling
• BabyLM 2026 evaluation and reproducibility studies
• Cognitive science research on language acquisition in non-English settings

Out-of-scope: Production use, sensitive applications, or tasks requiring high factual accuracy. Outputs are unfiltered and the model is trained on limited, machine-translated data.

---

Limitations

• Training data is machine-translated from English via IndicTrans2 and may carry translation artefacts or unnatural phrasing
• The strict data budget limits coverage of rare vocabulary and complex syntax
• Curriculum ordering is by byte length — a surface-level proxy for linguistic complexity; morphosyntactic difficulty in Telugu may not correlate perfectly with sentence length
• No RLHF or alignment fine-tuning

---

Citation

@misc{babylm_telugu_2026,
  author    = {pulipakav-1},
  title     = {babylm\_telugu\_2026: GPT-2 Language Models for Telugu with Random and Curriculum Training},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/pulipakav-1/babylm_telugu_2026}
}

@misc{babylm2026,
  title  = {BabyLM Turns 4: Call for Papers for the 2026 BabyLM Workshop},
  author = {BabyLM Team},
  year   = {2026},
  url    = {https://arxiv.org/abs/2602.20092}
}

@article{gala2023indictrans2,
  title  = {IndicTrans2: Towards High-Quality and Accessible Machine Translation of All 22 Scheduled Indian Languages},
  author = {Gala, Jay and others},
  year   = {2023},
  url    = {https://arxiv.org/abs/2305.16307}
}

---

Contact

Maintained by @pulipakav-1.