---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- 1.58-bit
- ternary
- cpu-1
- ablation
---

# CPU-1 Ablation Study — Source Checkpoints (compact 2-bit)

**Repo:** `Cukinator/cpu1-ablation-checkpoints` | **Unpacked mirror:** [`Cukinator/cpu1-ablations-final`](https://huggingface.co/Cukinator/cpu1-ablations-final) | **Code:** [`github.com/Cukinator/1.58bits`](https://github.com/Cukinator/1.58bits)

This repository stores the raw training checkpoints produced by `train_ablation_amd.py` from the 1.58bits repo. There are two checkpoint flavours, both saved per run inside its own folder:

| Filename pattern | Format | Purpose |
|---|---|---|
| `<run>/checkpoint_<run>_final.pt` | `compact_2bit` (2-bit packed ternary + bf16 scales) | Final inference checkpoint — minimal size, ~9 MB for a 39M ternary model |
| `<run>/checkpoint_<run>_step<N>.pt` | bf16 model + bf16 optimizer state | Phase 1 intermediate resume points |
| `<run>/checkpoint_<run>_phase2_step<N>.pt` | bf16 model + bf16 optimizer state | Phase 2 intermediate resume points (delete-gate runs only) |

> **If you just want ready-to-use float32 weights**, use the unpacked mirror at [`Cukinator/cpu1-ablations-final`](https://huggingface.co/Cukinator/cpu1-ablations-final) — those are plain `.pt` files you can load with `model.load_state_dict(...)` without any unpacking step.

This source repo exists so that (a) training jobs can resume from the latest step checkpoint after preemption, and (b) the `compact_2bit` format itself can be inspected and benchmarked.

## Repository contents

22 trained runs, organised in three rounds:

| Round | Tokens/param | Runs |
|---|---|---|
| **r1** — original ablation budget | 2 | `run_01`, `run_02`, `run_02a_byte_only_heads`, `run_03`, `run_04`, `run_05`, `run_05b_kernel_strict`, `run_06`, `run_07`, `run_08`, `run_09`, `run_10`, `run_13`, `run_14`, `run_15`, `run_16` |
| **r2** — re-run at higher budget | 15 | `run_04_r2`, `run_07_r2` (partial), `run_13_r2`, `run_14_r2`, `run_15_r2`, `run_16_r2` |
| **r3** — cold-start rescue (queued) | 50 | `run_04_r3`, `run_07_r3`, `run_14_r3`, `run_15_r3` (not yet uploaded) |

The naming and architecture of each run is defined in `RUN_CONFIGS` / `SMALL_RUN_CONFIGS` in `train_ablation_amd.py`.

## Quick start (compact_2bit)

Loading a `compact_2bit` checkpoint requires the unpacking helper that ships with the training code:

```python
import sys
sys.path.insert(0, "/path/to/1.58bits")
from train_ablation_amd import load_ablation_checkpoint, build_ablation_model, generate
import torch

state, config = load_ablation_checkpoint(
    "run_02/checkpoint_run_02_final.pt"
)
model = build_ablation_model(config)
model.load_state_dict(state, strict=False)
model.eval()

print(generate(model, "Once upon a time", 128, config, torch.device("cpu")))
```

For the same checkpoint without an external dependency, use the `cpu1-ablations-final` repo.

## Final-checkpoint sizes (compact_2bit)

Sizes are measured from the actual `_final.pt` files on disk.

| Run family | Architecture | d_model | Final size |
|---|---|---|---|
| `run_01` | Transformer + BPE (16K vocab) + FP16 | 512 | ~210 MB |
| `run_02`, `run_02a`, `run_03` | FP16 byte-level baselines | 512 | ~75 MB |
| `run_04`..`run_10` | 39M ternary chain | 512 | ~9 MB |
| `run_05b_kernel_strict` | MLGRU without W_o | 512 | ~8 MB |
| `run_13` | 10M BPE + ternary (4K vocab) | 320 | ~5 MB |
| `run_14`, `run_15`, `run_16` | 10M byte + ternary variants | 320 | ~3 MB |

## Training results & 2026-05 Audit

The full table of `val_loss`, `perplexity`, throughput and architecture per run is published in the README of the main repository. A summary of the 2026-05 audit findings:

1. **FP16 baselines** (`run_01`, `run_02`, `run_02a`, `run_03`) converge as designed: byte + LocalByteDecoder reaches `val_loss` 1.71, MLGRU FP16 reaches 1.86.
2. **All byte-level ternary runs collapse** to `ln(256) ≈ 5.545` nats — the uniform-output entropy floor. This holds across both scales (10M and 39M) and both token budgets (2 tok/p and 15 tok/p).
3. A **7.5× increase in tokens-per-parameter (r2)** moved the validation loss by only ~0.0001 nats. The cold-start dynamics of straight-through-estimator ternary training, not the budget, are the bottleneck at this scale.
4. Details, mechanistic analysis and throughput projections are documented in the main repository README.

## License
Apache-2.0. Same as the source code at [github.com/Cukinator/1.58bits](https://github.com/Cukinator/1.58bits).