NAME0x0 commited on
Commit
a3c0161
·
verified ·
1 Parent(s): f9aabd0

card: honest LiveCodeBench v6 result (v3@2167 -10pp vs donor)

Browse files
Files changed (1) hide show
  1. README.md +16 -2
README.md CHANGED
@@ -58,8 +58,22 @@ same donor tasks (mid-training, subset noise ≈ ±7pp):
58
  base model's ceiling; the specialization payoff is expected on *agentic / edit /
59
  harder* coding tasks, which are being added to the eval + training next.
60
 
61
- So: **donor-level coding + more reasoning, in a 2.6 GB file that runs on a 4 GB
62
- laptop.** A checkpoint, not a destination.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
  ## Files
65
 
 
58
  base model's ceiling; the specialization payoff is expected on *agentic / edit /
59
  harder* coding tasks, which are being added to the eval + training next.
60
 
61
+ **Harder held-out eval** (LiveCodeBench v6 competitive programming, 2025 contests
62
+ after the easy sets saturated; n=50 stdin problems, greedy, non-thinking):
63
+
64
+ | | Donor (Qwen3.5-4B) | This preview @ 2,167 |
65
+ |---|---|---|
66
+ | pass@1 | 44.0% | **34.0%** |
67
+
68
+ On fresh competitive-programming problems this checkpoint is currently **~10 pp
69
+ behind the donor** — the loss is on medium/hard problems (easy holds ~94%). Shown,
70
+ not hidden. This is the *expected direction* at 11% of training on off-distribution
71
+ data: the current mix is reasoning + edits, not competitive stdin, so basic SFT
72
+ hasn't helped (and slightly hurts) this slice yet. The number that matters is the
73
+ **trajectory** across later checkpoints, not this single mid-training point.
74
+
75
+ So: **more reasoning, donor-level on easy code benchmarks, but still behind the donor
76
+ on harder held-out coding — at 11% of training.** A checkpoint, not a destination.
77
 
78
  ## Files
79