Flow-Drafter-9B-v2

Joint-VAE tree drafter for Qwen/Qwen3.5-9B. One flow pass produces K future hidden states, which are expanded into a draft tree and verified in a single forward pass — accepting the longest path the target model agrees with, so decoding stays lossless (bit-exact at temperature 0).

v2 = v1 architecture retrained on an expanded, more diverse data mix (10 sources instead of 5: the original technical set plus ultrachat, opus_translation, no_robots, writingprompts, cnn_dailymail). No architecture change; drop-in replacement for Flow-Drafter-9B.

What changed

Tree-accept (tokens accepted per verify pass), v1 and v2 measured identically:

domain	v1	v2	Δ
HumanEval	5.12	5.10	-0.02
math_reasoning	5.77	5.54	-0.23
qa	3.18	3.31	+0.13
summarization	2.78	3.13	+0.35
writing	2.87	3.07	+0.20
mean	3.94	4.03	+0.09

The gain is concentrated on the free-form domains (summarization +0.35, writing +0.20, qa +0.13), which is what the diverse mix targets; math/code are flat-to-slightly-down as capacity reallocates, while staying strong in absolute terms. The same pattern was observed at 4B and 27B.

Read the numbers carefully. These come from an offline differential (scripts/diff_plugin_vs_harness.py): 5 bench domains, 200 windows each, generated tokens only, tree top_b=8, max_nodes=8, max_depth=5, K=8. They are not comparable to the canonical RedHatAI speculator-benchmark figures on the v1 card, which use a 64-node/depth-8 tree and read higher. The v1 row here is re-measured under this configuration so the comparison is apples-to-apples.

Deployment note

Under vLLM the achievable accept is lower than the offline numbers above, for two structural reasons: the served tree is smaller (16 nodes), and vLLM cannot supply the hidden state of the just-committed token at draft time, costing 0.7 accept. Measured end-to-end at 9B, batch 1, 7 domains: **1.1x** over stock vLLM, lossless. Speculative decoding pays off in proportion to base decode cost — 27B sees ~1.4x, 4B does not clear break-even.

Files

• model.safetensors — drafter weights
• chained_flow_tree_config.json — drafter config
• vae/ — the jointly-trained hidden VAE (unfrozen during drafter training, optimised for acceptance rather than reconstruction, with a reconstruction anchor keeping the latent decodable)