AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-NVFP4

Gemma-4-12B-it AEON Abliterated — K=4 Biprojection (NVFP4, MLP-only)

A quality-preserving 4-bit NVFP4 quantization of our K=4 biprojection abliteration of google/gemma-4-12B-it. Attention stays BF16, the MLP is NVFP4 — a deliberate split that keeps reasoning intact (within ~3-4pp of BF16 across MMLU/HumanEval) while delivering the throughput + memory win that 4-bit gives on bandwidth-bound hardware like the DGX Spark. ~11.7 GB, 254 tok/s. Loads in vLLM with --quantization modelopt.

Want zero measurable loss? Use the FP8 sibling — imperceptible from BF16, 13 GB. Want the smallest/fastest 4-bit with a reasonable trade-off? You're in the right place.

Refusal behavior has been removed; the model responds to prompts the base would decline. Operator-side safety is your responsibility — see the arbitration clause at the bottom.

---

🚀 QuickStart

Complete copy-paste recipe for the DGX Spark / Blackwell (Docker, recommended). Plain decode — no speculative drafter: on the GB10, 12B MTP is net-neutral (per-step drafter forward + 262k-vocab lm_head + multi-position verify roughly cancels the acceptance win), so this card runs straight decode. No drafter pull, no --speculative-config.

# 1. Pull the unified container (vLLM 0.24.0, sm_121a-native)
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest

# 2. Download this model (fresh)
huggingface-cli download AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-NVFP4 \
  --local-dir ./aeon-model

# 3. (No drafter — plain decode on the GB10.)

# 4. Serve (mount the model; vetted flags)
docker run -d --name aeon-gemma12b --gpus all --ipc=host --shm-size=16g --net=host \
  -v ./aeon-model:/model:ro \
  --entrypoint vllm \
  ghcr.io/aeon-7/aeon-vllm-ultimate:latest \
  serve /model \
    --served-model-name gemma12b \
    --quantization modelopt \
    --attention-backend triton_attn \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --enable-auto-tool-choice \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len 8192 \
    --max-num-seqs 16 \
    --gpu-memory-utilization 0.70 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code

# 5. Call (OpenAI-compatible)
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma12b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":128}' \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])"

Plain vLLM (pip)

# Container above ships vLLM 0.24.0 (sm_121a). Plain pip needs vLLM ≥ 0.22.2 (loader floor).
pip install "vllm>=0.22.2" "nvidia-modelopt>=0.43" "transformers>=5.10"
vllm serve AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-NVFP4 \
  --quantization modelopt --attention-backend triton_attn \
  --reasoning-parser gemma4 --tool-call-parser gemma4 --enable-auto-tool-choice \
  --enable-chunked-prefill --enable-prefix-caching \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 8192 --max-num-seqs 16 \
  --gpu-memory-utilization 0.70 --trust-remote-code

Unified-memory note: On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes. Discrete-VRAM GPUs can run higher.

⚠️ Needs vLLM ≥ 0.22.2 (for the Gemma4UnifiedForConditionalGeneration loader) and a Blackwell GPU (DGX Spark GB10 sm_121a, B100/B200, RTX 50-series). On Hopper/Ampere the FP4 weights dequantize with no speed benefit — use the BF16 sibling or FP8 there.

That's it. Everything below is detail.

---

Performance — DGX Spark (v0.24.0, aeon-vllm-ultimate:latest)

Measured on a single DGX Spark (GB10 / Blackwell sm_121a) under ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-07-01-v0.24.0; rollback :2026-06-18-v0.23.0-dflashfix) — vLLM 0.24.0 built from source for sm_121a. This is a dense 12B run with plain decode (no speculative drafter): on the GB10, MTP is net-neutral-to-slower (per-step drafter forward + 262k-vocab lm_head + multi-position verify roughly cancels the acceptance win), so this card runs straight decode and leans on concurrency throughput and the quant-vs-speed tradeoff instead.

Headline: ~15.8 tok/s single-stream decode, scaling to ~794 tok/s aggregate at c=64.

Single-stream (c=1) by category

Plain decode, FP8 KV cache, greedy. DFlash acceptance is n/a here — there is no drafter (plain decode).

Category	Decode tok/s	TTFT (ms)	TPOT (ms)	Prefill (tok/s)	DFlash accept
Coding	15.7	139	63.6	359	n/a
Math	15.7	140	63.8	472	n/a
Reasoning	15.8	136	63.4	383	n/a
Prose	15.7	138	63.7	291	n/a
Natural language	15.7	135	63.8	326	n/a
Extraction / JSON	16.1	141	62.0	405	n/a

Single-stream decode is memory-bandwidth-bound on the GB10's unified LPDDR5X pool — the ~15.7–16.1 tok/s band is essentially flat across categories because at c=1 the bottleneck is moving weights, not compute.

Aggregate throughput by concurrency

Where the NVFP4 build earns its keep. Aggregate decode tok/s climbs near-linearly with concurrency and peaks at c=64 (Prose, the best category):

Concurrency	Coding	Math	Reasoning	Prose	Natural language	Extraction / JSON
c=1	16	16	16	16	16	15
c=8	137	137	114	137	137	117
c=16	259	257	242	260	259	236
c=32	471	465	382	476	474	428
c=64	779	766	755	794	789	687

The DFlash high-concurrency fix in this image (block-table slice to the unpadded batch) is what lets the server scale cleanly to c=64 — the prior image crashed at c≥32 under speculative decoding. Even on this plain-decode card, the fixed image is what carries the c=64 sweep without engine death.

Long-context draft acceptance is n/a for this card — plain decode, no drafter — so no long-context acceptance curve is reported.

Quant family — pick your point on the curve

The same dense 12B, four ways. Single-stream is bandwidth-bound (all near ~15–16 tok/s except BF16); the real spread is in size and aggregate throughput:

Variant	c=1 decode	c=64 aggregate	Notes
NVFP4 MLP-only (this)	16	794	11.7 GB; minimal-loss 4-bit
FP8	16	747	13 GB; near-lossless
NVFP4-FP8 (mixed)	21	917	9.3 GB; fastest + smallest — supersedes this
BF16	8	458	24 GB; reference precision

The 4-bit NVFP4 weights cut size ~51% vs BF16 and push c=64 aggregate from 458 → 794 tok/s; the Mixed NVFP4+FP8 sibling is the Pareto winner (smaller and faster). Single-stream BF16 is ~2× slower because, at c=1, decode is gated on weight bandwidth and BF16 moves twice the bytes.

What we fixed for the DGX Spark

The whole fleet runs on one unified container — ghcr.io/aeon-7/aeon-vllm-ultimate:latest — vLLM 0.24.0 built from source for sm_121a and merged with the AEON speculative-decoding stack. The two fixes most relevant to this card:

• Unified container (vLLM 0.24.0, sm_121a-native). One image for the whole fleet, compiling the SM120-family CUTLASS NVFP4/FP8 kernels the GB10 actually dispatches to — true 4-bit tensor-core throughput rather than dead B200-only kernels, plus the boot/CUDA-graph patches needed to start on a GB10.
• DFlash high-concurrency fix. Slices the speculative drafter's KV block-table to the unpadded batch (block_table[:num_reqs]); the drafter previously crashed at ≥32 concurrent requests (a padded-vs-unpadded block-table shape mismatch in FlashAttention). It now scales cleanly to c=64 — a port of upstream PR #43982, which fixed this for MTP but never for DFlash. This card runs plain decode, but the fix is what keeps the c=64 sweep above from killing the engine.

Stock baseline: no fresh fully-vanilla (stock upstream vLLM, no AEON / sm_121a opts) baseline exists for this model yet — a same-harness vanilla re-bench is pending. The figures above are all measured on aeon-vllm-ultimate:latest (vLLM 0.24.0).

---

Capability — full-length eval (measured)

All four axes via the vLLM serving path, identical prompts/settings. MMLU is balanced across all 57 subjects (5 each = 285 Q) — not a single-subject slice. HumanEval is the full 164 problems.

Model	MMLU (285)	HumanEval-syn (164)	HumanEval-fun (164)	IFEval (50)
google/gemma-4-12B-it (official base)	81.4%	99.4%	82.9%	90%
K4-BF16 (abliterated, full precision)	80.4%	99.4%	83.5%	90%
K4-NVFP4 MLP-only (this)	76.8%	96.3%	76.2%	90%
K4-FP8 (sibling, near-lossless)	80.4%	99.4%	85.4%	90%

The 4-bit trade-off is small and honest: −3.6pp MMLU, −3.1pp HumanEval-syntactic, −7.3pp HumanEval-functional vs the BF16 model; instruction-following (IFEval) is unchanged. This is a minimal-loss result for a 4-bit quant — achieved by keeping all attention layers at BF16 (where precise multi-step reasoning lives) and quantizing only the MLP (bandwidth-bound, FP4-friendly). For comparison, a naive full-W4A4 NVFP4 of this model loses ~21pp on hard reasoning; this MLP-only split avoids that.

Throughput (DGX Spark GB10, FP8 KV cache, greedy)

	NVFP4 MLP-only (this)	BF16	FP8
Concurrent ×16 aggregate	254 tok/s	144 tok/s	235 tok/s
Single-stream overall	15.7 tok/s	7.7 tok/s	15.8 tok/s
Size	11.7 GB	24 GB	13 GB

1.76× BF16 throughput at under half the memory — the win that matters on the DGX Spark's unified-memory GB10, where weight bandwidth dominates decode. KV cache holds ~1.23M tokens at 8k ctx (≈150× max concurrency).

Quantization methodology

Property	Value
Tool	NVIDIA ModelOpt 0.43.0
Config	NVFP4_MLP_ONLY_CFG
What's quantized	MLP gate_proj / up_proj / down_proj on all 48 layers → NVFP4 (E2M1, block_size=16, E4M3 block scales)
What stays BF16	all self_attn (q/k/v/o), lm_head, embed_tokens, embed_vision* / embed_audio* / vision_embedder*
Why MLP-only	Gemma-4 attention carries the precise-reasoning signal + per-channel outliers; keeping it BF16 preserves MMLU/reasoning. NVIDIA ships their Gemma-4-31B-IT-NVFP4 the same way.
Calibration	1024 × CNN/DailyMail validation @ 1024 tokens, native sm_121a
Size	~11.7 GB (from 23.9 GB BF16 — 51% reduction)
Runtime	vLLM --quantization modelopt via Gemma4UnifiedForConditionalGeneration

vLLM loader note (reproducers)

Google's Gemma-4-12B is the encoder-free Gemma4UnifiedForConditionalGeneration. ModelOpt's HF export needs two touch-ups to load in vLLM: rename the vision keys to vLLM's vision_embedder.* layout, and add model.vision_embedder* to the quant ignore list. Scripted in make_vllm_ready.py. Requires vLLM ≥ 0.22.2.

Abliteration methodology

K=4 multi-direction norm-preserving biprojection (extends TrevorJS). Basis layers L24/L37/L39/L26, o_proj + mlp.down_proj edited on 24/48 layers, scale=1.0. The eval above shows the abliteration is capability-neutral — K4-BF16 is within ~1pp of Google's official base on every axis. Full math on the BF16 card.

Available formats

Variant	Repo	Precision	Size	Pick when
FP8	…-K4-FP8	FP8 E4M3	13 GB	Quality matters — near-lossless, matches BF16
Mixed NVFP4+FP8	…-K4-NVFP4-FP8	NVFP4 MLP + FP8 attn	9.3 GB	Smallest + fastest — MLP-only quality, 20% less size, 34% faster
NVFP4 MLP-only (this)	…-K4-NVFP4	NVFP4 MLP + BF16 attn	11.7 GB	Superseded by Mixed NVFP4+FP8 (above)
BF16	…-K4-BF16	bfloat16	24 GB	Fine-tuning, non-Blackwell hardware

Acknowledgements

TrevorJS (biprojection), p-e-w/heretic (framework), NVIDIA ModelOpt (NVFP4 toolkit + Gemma-4-31B MLP-only reference recipe), AEON-7 (K-direction extension, MLP-only NVFP4 recipe + vLLM loader fixes + capability eval).

License

Inherits the Gemma license.

---

Arbitration Clause

By accessing, downloading, using, running inference on, fine-tuning, merging, quantizing, distributing, integrating, or otherwise interacting with this model, you acknowledge and agree to the following:

1. Sole Responsibility. You, the user, are solely and exclusively responsible for (a) every prompt you or your downstream system issue to this model, (b) every response this model produces in reply, (c) every downstream action taken by you, your systems, your agents, or your users in reliance on those responses, and (d) any harm — direct, indirect, consequential, foreseeable, or otherwise — that results from any of the above.

2. No Warranty. This model is provided strictly "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, non-infringement, safety, alignment, factual accuracy, or legal compliance in any jurisdiction. No contributor, author, publisher, or hosting platform assumes liability of any kind for outputs or downstream use.

3. Legal Compliance. You are responsible for ensuring that your use of this model complies with all applicable laws, regulations, terms of service, industry codes of conduct, professional ethical standards, and organizational policies in every jurisdiction in which you operate or in which your outputs may be received. The unaligned nature of this model does not grant you any legal authorization you did not already have.

4. Operational Safety Layer. An uncensored model is not a toy. You are expected to implement appropriate downstream safety layers proportionate to your deployment context, including but not limited to: input validation, output filtering, content moderation, audit logging, rate limiting, access controls, and human-in-the-loop review for high-risk workflows. A production deployment of this model without such layers is unsafe by construction and is not a supported use case.

5. Heightened Duty of Care. The absence of internal refusal behavior means the duty of care that would ordinarily rest partly with the model rests entirely with you. You are expected to exercise greater — not lesser — caution, forethought, and ethical discipline when operating this model than you would operate a base aligned model. If you are uncertain whether your contemplated use is ethical, legal, or wise, the correct action is to not make the request.

6. No Endorsement of Outputs. The authors, contributors, and publishers of this model do not endorse, adopt, or take responsibility for any specific output this model produces. Outputs are a stochastic function of the prompt, the weights, and the sampler state — not a statement of position by any human.

7. Arbitration. Any dispute, claim, or controversy arising out of or relating to the use of this model, its outputs, or this clause shall be resolved through binding individual arbitration under the rules of a mutually agreed arbitration body (or, absent agreement, the American Arbitration Association's Consumer Arbitration Rules), waiving any right to a jury trial, class action, representative action, or consolidated proceeding. Venue shall be the jurisdiction of the disputing party bringing the claim. Costs and attorneys' fees shall be allocated per the applicable arbitration rules. This clause does not expand, and where legally prohibited does not establish, any liability in the other direction; it limits how the user may proceed when alleging harm tied to their own use of this model.

8. Indemnification. You agree to indemnify, defend, and hold harmless the authors, contributors, and publishers of this model from and against any claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from or related to your use of the model or your breach of this clause.

9. Severability. If any provision of this clause is held unenforceable in a given jurisdiction, the remaining provisions remain in full force in that jurisdiction, and the unenforceable provision is replaced by the closest enforceable equivalent consistent with the original intent.

10. Acceptance. Your use of this model constitutes your acceptance of this clause in full. If you do not accept, do not use the model.

This model is a tool with no opinions of its own. You supply the opinions. You supply the judgement. You supply the ethics. The outputs carry your fingerprints, not the model's.

---

☕ Support the work

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

₿ Bitcoin (BTC) bc1q09xmzn00q4z3c5raene0f3pzn9d9pvawfm0py4

Ξ Ethereum (ETH) 0x1512667F6D61454ad531d2E45C0a5d1fd82D0500

◎ Solana (SOL) DgQsjHdAnT5PNLQTNpJdpLS3tYGpVcsHQCkpoiAKsw8t

ⓜ Monero (XMR) 836XrSKw4R76vNi3QPJ5Fa9ugcyvE2cWmKSPv3AhpTNNKvqP8v5ba9JRL4Vh7UnFNjDz3E2GXZDVVenu3rkZaNdUFhjAvgd

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.