amosluna/babylm-2026-strict-small-mdlm-seed42

Masked-Diffusion BabyLM (Strict-Small) — MD_base seed 42

BabyLM 2026 Strict-Small (English) submission checkpoint. A LLaDA/MDLM-style absorbing-state masked-diffusion language model on a GPT-2-scale bidirectional Transformer, trained on <=10M unique words for <=10 epochs (<=100M words seen).

• Method: MD_base (masked diffusion)
• Seed: 42
• Words seen (final): 99996737
• Intermediate checkpoints: 19 branches chck_1M ... chck_100M

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("amosluna/babylm-2026-strict-small-mdlm-seed42")
model = AutoModelForMaskedLM.from_pretrained("amosluna/babylm-2026-strict-small-mdlm-seed42", trust_remote_code=True)
# Intermediate checkpoint:
# model = AutoModelForMaskedLM.from_pretrained("amosluna/babylm-2026-strict-small-mdlm-seed42", revision="chck_10M", trust_remote_code=True)

Evaluation

A masked-diffusion denoiser is scored like a masked LM (per-token pseudo-log-likelihood), so the official BabyLM pipeline evaluates it with the mlm backend:

cd strict
./eval_zero_shot.sh amosluna/babylm-2026-strict-small-mdlm-seed42 mlm
./eval_finetuning.sh --model_path amosluna/babylm-2026-strict-small-mdlm-seed42 --seed 42
bash scripts/collate_preds.sh amosluna/babylm-2026-strict-small-mdlm-seed42 mlm strict-small --fast

See the project's docs/EVALUATION.md for the diffusion-native ELBO scorer and the inference-time layer-duplication variant.