DataBench

Rather than training one model, we trained multiple models, each on a different dataset. Everything else was kept the same, so that the only changing variable would be the dataset itself.

Model Architecture

• Base Architecture: LlamaForCasualLM
• Tokenizer: Harley-ml/Dillionv2-1.3M
• Transformers Version: 5.13.1
• Hidden Size: 128
• Vocab Size: 2564
• Number of Layers: 8
• Number of Heads: 4
• Number of KV Heads: 2
• Intermediate Size: 344
• Head Dim: 32
• Max Position Embeddings: 256
• RoPe Theta: 2500.0
• Tie Word Embeddings: true
• Hidden Activation: silu
• MLP Bias: false
• Initializer Range: 0.2
• RMS Norm Eps: 1e-06
• Pretraining Tp: 1
• Use Cache: false
• Total Parameters: 1,780,352

Training Setup

• Epochs: 1
• Max Steps: -1.0
• Batch Size: 400
• Sequence Length: 256
• Gradient Accumulation: 2
• Gradient Clipping: 1.0
• Gradient Checkpointing: true
• Learning Rate: 2.5e-3
• Eval Split: 0.00165
• Weight Decay: 0.01
• Optimizer: AdamW
• AdamW Betas: (0.9, 0.95)
• AdamW Eps: 1e-8
• Scheduler: WSD
• WSD Warmup Ratio: 0.015
• WSD Stable Ratio: 0.78
• WSD Decay Ratio: 0.20
• WSD Minium LR Ratio: 0.0
• WSD Number of Cycles: 0.5
• DType: float16
• Torch.Compile: true
• DataLoader Workers: 2
• Seed: 311

Results

Accuracy is normalized by length and shown as a percentage.

Dataset	ARC-Easy	HellaSwag	PIQA	Avg ↑
FineWeb	27.82%	26.89%	52.83%	35.85%
DCLM-1.0-Baseline	29.08%	27.06%	52.23%	36.12%
DOAB	31.14%	28.00%	52.18%	37.11%
Project Gutenberg	25.84%	24.63%	50.21%	33.56%
EOT-2004-Raw	29.08%	27.54%	51.31%	35.98%
Wikipedia	28.28%	27.61%	51.14%	35.68%

(Will add ArithMark-3.0 later)

Notice

This is a work in progress and is currently not completed. By the end of this project, we aim to have tested over 50 datasets.

License

Apache 2.0.