DABL — Dual-SSM Attention-Bridge Learning (research checkpoints)
Trained Mamba-2 / SSD student checkpoints from a study of whether softmax
attention's normalizer (Σexp numerator / Σexp denominator) can be
transplanted into an SSM student distilled from a Transformer teacher.
Answer: no. On fair held-out data (train on C4, evaluate on WikiText-2 test), every softmax-faithful normalizer variant ties or loses to a plain normalizer-free single-SSM. This is a negative-result artifact release.
- Code / full experimental record: https://github.com/tlstngud/mangochange
- Teachers:
microsoft/phi-1_5(MIT),Qwen/Qwen3-0.6B(Apache-2.0),TinyLlama(Apache-2.0) - Distillation: 4-stage MOHAWK-style (matrix orientation → hidden-state / bridge → end-to-end KD).
Headline (WikiText-2 test PPL; C4 training, seq1024, 300M tokens, Phi-1.5 teacher)
| variant | normalizer | PPL |
|---|---|---|
| single | none | 335.31 |
| sig_gate | fixed mass, opt-in | 335.49 |
| sieber | learned η(x)=exp(Wx) | 334.39 |
| soft_floor | signed row-sum ÷ | 382.42 |
| deltanet | L2-key kernel | 391.63 |
| l1 | L1 mass ÷ | 395.64 |
| hedgehog | learned feature maps | 403.71 |
single, sig_gate, and sieber cluster within ~1 PPL — whether the normalizer
stream is absent, fixed, or learned (the Sieber et al. NeurIPS 2024 upgrade),
it adds nothing on fair data.
Checkpoints
Each folder under checkpoints/ holds a run's student.pt (raw PyTorch
state dict) and its results.json (config + eval metrics). Folder names encode
the setting, e.g. sieber_c4_1024_300M, single_hd256_bs1_c4_300M,
dabl_phi_300M_c4. Load with the model definition in the GitHub repo
(dual_mamba/dabl_v4.py); the matching --denom-mode / dims are in each
results.json.
Intended use
Research reproducibility and analysis of Transformer→SSM distillation and attention-normalizer transplantation. Not tuned for downstream deployment.