JasonW2025/Laguna-S-2.1-ModelOpt-NVFP4-W4A4-vllm

Laguna-S-2.1 — NVIDIA ModelOpt NVFP4 W4A4, 4.70 bpp

A 64.38 GiB NVFP4 quantization of poolside/Laguna-S-2.1 (117.6 B params) that serves on a single DGX Spark (GB10, sm_121a).

Weights and activations are 4-bit on the MoE experts — not weight-only. That matters on this hardware: it is what lets the fused NVFP4 CUTLASS MoE kernel run at all. A weight-only NVFP4 export has a QuantKey ending xNone, which every fused backend rejects, leaving only MARLIN.

weights on disk	69,143,730,240 B = 64.38 GiB (7 shards, 145,441 tensors)
average	4.70 bits/parameter
agentic tool-use, full-69	87 and 88 (n=2) — read Quality before quoting

Where the bytes actually are

Measured by reading every tensor's dtype and byte range out of the shard headers, not inferred from config_groups:

component	GiB	%	stored as
routed experts (layers 1–47)	59.485	92.4 %	NVFP4 W4A4 — uint8 packed pairs + e4m3 block scales + fp32 global/input scales
attention q/k/v/o (48 layers)	2.620	4.1 %	FP8 W8A8 (e4m3 + fp32 scales)
mlp.shared_expert (47 layers)	0.826	1.3 %	BF16
embed_tokens	0.574	0.9 %	BF16
lm_head	0.574	0.9 %	BF16
layer-0 dense MLP	0.211	0.3 %	BF16
router + e_score_correction_bias	0.069	0.1 %	BF16
norms / other	0.001	0.0 %	BF16
total	64.359

Read the 4.70 bpp headline with that table in hand. The experts alone are ~4.50 bpp; the rest of the average is BF16 embeddings, lm_head, shared expert and layer-0 MLP — 1.185 GiB of it in embed_tokens + lm_head for a 100,352-token vocabulary. If you are comparing this against a lower-bpp quantization of the same model, compare the expert format, because that is where 92.4 % of the bytes live and where the two will differ.

config_groups lists only 192 + 47 targets — 48×4 attention projections and 47 expert stacks. Everything else in the table above is untouched BF16.

---

⚠️ Serving: three things that will bite you

1. The KV cache is FP8 with no calibrated scales, and auto won't change that

config.json declares kv_cache_scheme: {"dynamic": false, "num_bits": 8, "type": "float"} — static FP8 KV — and the checkpoint ships zero k_scale/v_scale tensors. We verified this directly: 0 of either in the weight map.

Serving FP8 KV against that quantizes the cache at scale 1.0 against a distribution nothing was calibrated to. On this model family that measured −2 points of agentic score for no speed gain. It is a defect in our export (a ModelOpt recipe used use_constant_amax: true, so _get_amax returned a constant, enable_stats_collection skipped the quantizer, no _amax buffer was ever created, and the exporter emitted nothing). We are disclosing it rather than quietly shipping it.

It is the default, and --kv-cache-dtype auto does NOT save you. vLLM reads kv_cache_scheme out of the quantization config before honouring auto, resolves to kv_cache_dtype=fp8_e4m3, and logs:

WARNING [kv_cache.py:134] Checkpoint does not provide a q scaling factor.
        Setting it to k_scale. This only matters for FP8 Attention backends

— the stack noticing the missing scales and continuing anyway. Every number on this card was measured in that state, because it is what the checkpoint does out of the box.

To actually get an unquantized KV cache you have to remove the declaration. Serve a config overlay rather than editing the checkpoint in place:

python3 -c "
import json, pathlib
c = json.loads(pathlib.Path('config.json').read_text())
c['quantization_config'].pop('kv_cache_scheme', None)
pathlib.Path('config.bf16kv.json').write_text(json.dumps(c, indent=2))
"
# then serve with config.bf16kv.json in place of config.json

We have not re-benchmarked under that overlay. At the context lengths in Performance the KV cache is small enough that its dtype should not move throughput measurably — the reason to do this is quality, not speed.

2. moe_backend must be a fused CUTLASS one on GB10

Two work, and they are the two you should choose between:

• FLASHINFER_CUTLASS — what the numbers in Performance were measured with.
• CUTLASS (vLLM's own, logged as VLLM_CUTLASS) — also runs on sm_121; we have served this artifact under it at 1,048,576 max-model-len.

Two do not work, and two are actively misleading:

• FLASHINFER_CUTEDSL and FLASHINFER_TRTLLM fail is_device_capability_family(100), which computes current.to_int() // 10 == capability // 10; GB10 is sm_121 → 12, so it matches family 120, not 100. Forcing past the guard gets a get_nvcc_flags_list( supported_major_versions=[10]) failure in CUTEDSL and a cubin named …_sm100f in TRTLLM.
• MARLIN and HUMMING load this checkpoint and silently ignore its 36,288 activation scales. vLLM's nvfp4 oracle nulls a13_scale/a2_scale on those paths and builds nvfp4_w4a16_moe_quant_config — the function name is the proof. Your W4A4 model then runs W4A16 numerics with no warning at all. If you benchmark this artifact under MARLIN you are not benchmarking W4A4.

Check the startup line Using '<X>' NvFp4 MoE backend out of potential backends: [...] and confirm <X> is one of the two fused CUTLASS options.

vllm serve <model> --served-model-name laguna-w4a4 \
  --tensor-parallel-size 1 --dtype bfloat16 \
  --attention-backend FLASHINFER \
  --kernel-config '{"moe_backend":"FLASHINFER_CUTLASS"}' \
  --gpu-memory-utilization 0.85 --max-model-len 32768 --max-num-seqs 4 \
  --kv-cache-dtype auto \
  --no-enable-prefix-caching \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1

Environment: CUTE_DSL_ARCH=sm_121a, VLLM_TRUST_REMOTE_CODE=1. Measured on vllm/vllm-openai:v0.25.1.

3. The reasoning parser is broken upstream (affects every Laguna deployment)

vLLM's PoolsideV1ReasoningParser derives from the DeepSeek-V3 parser, which picks its implementation from chat_template_kwargs["thinking"] / ["enable_thinking"]. Both default to False, so it silently becomes an IdentityReasoningParser whose reasoning_start_str/reasoning_end_str are None — vLLM logs "Auto-initialization of reasoning token IDs failed" and carries on. Chain-of-thought then lands in scored content and a stray </think> appears in ~68 of 69 bench scenarios.

Every score on this card was measured with that defect present, so they are all a few points low. On a sibling artifact, fixing it was worth +3. We have not re-run this one.

---

Performance

One DGX Spark, GB10 / sm_121a, 121.63 GiB unified. vllm/vllm-openai:v0.25.1, the serve line above, FLASHINFER_CUTLASS confirmed in the log. Strictly sequential (concurrency 1), temperature 0, prefix caching off, warmups discarded, median of 3. KV cache 1,200,440 tokens at --max-model-len 32768 --gpu-memory-utilization 0.85.

prompt tok	output tok	TTFT	prefill	decode	e2e
78	256	909.8 ms	—	28.00	25.56
78	512	219.8 ms	—	27.92	27.65
636	8	858.6 ms	727 tok/s	27.88	7.21
2 351	8	1 174.8 ms	2 034 tok/s	27.73	5.61
9 356	8	2 559.2 ms	3 656 tok/s	27.51	2.84
18 749	8	5 705.6 ms	3 283 tok/s	27.72	1.34
2 351	256	767.3 ms	3 100 tok/s	27.67	25.65

Decode: 27.96 tok/s (median; 27.51–28.00 across every row). Note how flat that is — decode on a top-10-of-256 MoE is bandwidth-bound on the active slice, so it barely moves with context at these lengths.

Prefill: ~3 300–3 660 tok/s once the prompt is large enough for compute to dominate. Below ~2 k tokens the "prefill" column is meaningless — TTFT there is fixed overhead, not throughput — so do not read the 727 tok/s row as a prefill rate.

TTFT, with its spread — because the spread is the finding

Warmups discarded per shape, 7 kept samples each:

prompt tok	median	min	max	spread
81	216.1 ms	214.7	217.1	2.4
186	698.7 ms	253.0	713.7	460.7
633	821.6 ms	316.4	827.0	510.6
2 366	1 141.8 ms	770.0	1 160.7	390.7
9 325	2 550.3 ms	2 539.9	2 567.1	27.2

The two ends are tight — ±1 ms at 81 tokens, ±14 ms at 9 325. The middle is bimodal: at 633 tokens a minority of requests complete in ~316 ms and the rest in ~825 ms. That is not warmup (discarded per shape) and we do not currently have an explanation for it. It is published as a range rather than smoothed into a median, because a single number here would misrepresent what you will actually observe.

Backend choice is worth ~6 % decode and ~25 % prefill

Same weights, same box, same day, only moe_backend changed:

	FLASHINFER_CUTLASS	CUTLASS (VLLM_CUTLASS)
decode	27.96 tok/s	26.46 tok/s
prefill peak	3 656 tok/s	2 922 tok/s

Both are correct W4A4 paths that honour the activation scales; FLASHINFER_CUTLASS is faster on this hardware. (The VLLM_CUTLASS column was taken at 1 M max-model-len with CUDA graphs on, so it is indicative rather than a controlled one-variable comparison.)

---

Quality — agentic tool use

69 scenarios covering tool selection, argument construction, multi-step planning, restraint and safety. Temperature 0, concurrency 1, prefix caching off, two runs in one serve session, scenario_count == 69 asserted on every run.

	W4A4 r1	W4A4 r2	poolside NVFP4 r1	poolside NVFP4 r2
final score	87	88	87	85
points	120/138	121/138	120/138	117/138

Per category (earned / max):

		W4A4 r1	W4A4 r2	poolside r1	poolside r2
A	Tool Selection	6/6	6/6	6/6	6/6
B	Parameter Precision	6/6	6/6	6/6	4/6
C	Multi-Step Chains	6/8	6/8	6/8	8/8
D	Restraint & Refusal	6/6	6/6	6/6	6/6
E	Error Recovery	6/6	6/6	6/6	6/6
F	Localization	6/6	6/6	6/6	6/6
G	Structured Reasoning	6/6	6/6	6/6	6/6
H	Instruction Following	6/10	6/10	6/10	6/10
I	Context & State	16/20	18/20	16/20	18/20
J	Code Patterns	6/6	6/6	6/6	5/6
K	Safety & Boundaries	21/26	20/26	22/26	19/26
L	Toolset Scale	8/8	8/8	6/8	6/8
M	Autonomous Planning	5/6	5/6	5/6	5/6
N	Creative Composition	4/6	6/6	5/6	6/6
O	Structured Output	12/12	10/12	12/12	10/12

The noise floor, stated up front

This artifact disagrees with itself. Across its own two runs — identical weights, identical server, same session — four of fifteen categories moved:

category	r1	r2	Δ
I Context & State	16	18	+2
N Creative Composition	4	6	+2
O Structured Output	12	10	−2
K Safety & Boundaries	21	20	−1

That spread is larger than most of the differences between arms in the table above. Treat per-category gaps of 1–2 points as noise unless they repeat.

The reason is not the bench harness. In a controlled study on this stack we found the server is not deterministic at temperature 0: the same request repeated returns different logits, only 20 of 103,752 scored positions are bitwise stable across replicates of one arm, and ~50 % of greedy replicates diverge within the first two tokens.

What we can and cannot claim about W4A4 specifically

We ran a dedicated 4-arm study to test whether 4-bit activations cost anything, using byte-identical weights (109,341 shared tensors verified identical by blake2b) served two ways — once under CUTLASS honouring the activation scales, once under MARLIN silently dropping them.

Every between-arm divergence sat inside the within-arm band. The activation-honouring arm's KL from the activation-dropping arm (0.266) was smaller than its KL from itself (0.308). Paired sign tests on the agentic bench gave p = 0.375 and p = 1.000, and 4 of the 5 discordant scenarios were ones that also flip within a single arm's own two runs.

So: do not read "W4A4 87/88 vs W4A16 81/86" as W4A4 being better. We have refuted that ordering three independent ways. What this artifact is warranted for is the prefill and composition properties in the Performance section, not a quality win.

No BF16 teacher exists

117.6 B parameters in BF16 is 235 GB; a Spark has 128. Nobody — us included — can currently report a KL or perplexity delta against the unquantized model on this hardware. Any claim that this quantization "matches" the original is unsupported, ours included. If you need a fidelity guarantee rather than a task score, measure it on your workload.

---

Provenance and licensing

• Base: poolside/Laguna-S-2.1, OpenMDW-1.1. LICENSE.md is included here as that license requires, along with the base model's configuration_laguna.py and modeling_laguna.py.
• Quantizer: NVIDIA TensorRT Model Optimizer (ModelOpt), NVFP4 W4A4 on the routed experts, FP8 W8A8 on attention and the shared expert.
• Verified before upload: 7/7 shards' declared end equals file size exactly; index resolves both ways over 145,441 tensors; 141 shared_expert keys and 0 shared_mlp — a sibling copy of this artifact carries the wrong (HY3) names and is quarantined, because serving it leaves 47 shared experts at random init with no error.

Quantized by Jason Wong. No affiliation with poolside or NVIDIA.