Huihui-Qwen3.6-27B-abliterated-NVFP4

English | 繁體中文

---

English

Quantized using llm-compressor with mixed-domain calibration and sensitive-layer protection for maximum accuracy recovery.

Native W4A4 on DGX Spark (SM121) — confirmed working

This checkpoint runs true W4A4 via FlashInfer CUTLASS NVFP4 kernel (verified in vLLM logs: FlashInferCutlassNvFp4LinearKernel). Requires:

• vLLM 0.19.1rc1.dev374+g1174723eb or later (includes PR #37725 arch-suffix fix)
• FlashInfer ≥ 0.6.8 with SM120f compilation (PR #2650)
• CUDA ≥ 12.9

MTP speculative decoding supported — use method: qwen3_next_mtp (dense-model MTP, distinct from the MoE mtp method).

NVFP4 W4A4 quantization of huihui-ai/Huihui-Qwen3.6-27B-abliterated, optimized for NVIDIA DGX Spark (GB10 SM121) with the FlashInfer CUTLASS FP4 kernel.

Model Details

Item	Value
Architecture	Dense (27B), 64 layers — 16 × (3 × Gated DeltaNet → FFN + 1 × Gated Attention → FFN)
Base model	Qwen/Qwen3.6-27B
Fine-tuned by	huihui-ai (abliteration)
Quantized by	YuYu1015
Model size	~28.6 GB (NVFP4 + BF16 MTP/GDN layers, vs ~71.9 GB BF16 original)
Context length	Up to 262,144 tokens
Thinking mode	Supported (enable_thinking: true/false)
Tool calling	Supported (qwen3_coder parser)
MTP	Built-in MTP weights (qwen3_next_mtp method)
DFlash	Compatible with z-lab/Qwen3.6-27B-DFlash

Quantization Details

This model uses a three-strategy stack (ACD) on top of the llm-compressor official flow:

Strategy	Description
A. Official baseline	Qwen3ForCausalLM + save_mtp_tensors_to_checkpoint (preserves MTP weights after quantization)
C. Mixed-domain calibration	ultrachat_200k (128 chat) + Nemotron-Post-Training-Dataset-v2 (128 reasoning) = 256 total
D. Sweet-spot hyperparameters	num_calibration_samples=256, max_seq_length=4096 (quality > quantity)

B (last-layer protection) incompatible with vLLM fused kernels: Partial ignore of linear layers within a transformer block triggers ValueError: All projections need to have same quantization scheme but found multiple.

E (SpinQuant R1+R2) incompatible with multi-modal config: llm-compressor's get_head_dim only reads top-level config, not Qwen3.6's nested text_config.

Item	Value
Method	llm-compressor (main) + compressed-tensors (main)
Scheme	NVFP4 W4A4 (E2M1 + FP8 per-group scaling, group size 16)
Format	compressed-tensors
Calibration datasets	HuggingFaceH4/ultrachat_200k (128) + nvidia/Nemotron-Post-Training-Dataset-v2 (128)
Calibration samples (total)	256
Calibration sequence length	4096
Hardware	NVIDIA DGX Spark (GB10, 128GB unified memory)
Environment	transformers>=5.0,<6 + llm-compressor main + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Layers Preserved in BF16

Layer pattern	Reason
re:.*lm_head	Output head, sensitive to quantization noise
re:.*embed_tokens$	Input embeddings
re:visual.* / re:model.visual.*	Vision encoder
re:.*mlp.gate$	FFN gate (dense model; must stay BF16)
re:.*linear_attn.*	Gated DeltaNet (Mamba) layers — may output zeros if quantized
mtp.* (all MTP weights)	Reattached in BF16 via save_mtp_tensors_to_checkpoint after quantization

Speculative Decoding

This model supports two speculative decoding methods:

MTP (built-in weights, recommended default for this abliterated variant — see warning at top):

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

DFlash (separate drafter, recommended for single-user / low-concurrency on non-abliterated variants):

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}'

Note: On hybrid GDN architectures, MTP may hit a state-rollback bug (vLLM #39273) during high token-rejection rates. Reduce num_speculative_tokens to 1 if you observe degraded output quality.

Serving with vLLM

vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3.6-27b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --attention-backend flash_attn \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 131072 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --performance-mode throughput \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) Compatibility Notes

• Native W4A4 confirmed via FlashInfer CUTLASS NVFP4 kernel
• Verify in logs: Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
• FP8 KV cache is not compatible with GDN non-causal attention; use --kv-cache-dtype auto
• Dense model fits more comfortably on 128 GB UMA than the 35B BF16 weight — higher max-model-len is achievable
• --language-model-only skips vision encoder profiling for text-only inference
• Clear page cache before starting on UMA: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Known Limitations

• Per-tensor global scales for fused q_proj / k_proj / v_proj may differ, causing a vLLM warning at load time. Impact on accuracy is typically small but measurable on strict tool-calling JSON schemas.
• DFlash drafter was trained on the original Qwen3.6-27B, not the abliterated variant — acceptance rate may be lower than on the original model.

Safety Warning

This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.

Credits

• Original Model: Qwen/Qwen3.6-27B by Alibaba Qwen Team
• Abliteration: huihui-ai
• NVFP4 Quantization: YuYu1015 on NVIDIA DGX Spark (GB10)
• Quantization Tool: llm-compressor by vLLM Project
• Reference Quantization: sakamakismile/Qwen3.6-27B-NVFP4
• Sensitivity Analysis: Diagnosing FP4 Inference (arXiv 2603.08747)

---

繁體中文

使用 llm-compressor 搭配混合領域校準與敏感層保護量化，最大化精度保留。

DGX Spark (SM121) 原生 W4A4 — 已驗證可用

此 checkpoint 透過 FlashInfer CUTLASS NVFP4 kernel 跑真 W4A4（vLLM log 可見 FlashInferCutlassNvFp4LinearKernel）。需要：

• vLLM 0.19.1rc1.dev374+g1174723eb 以上（含 PR #37725 arch-suffix 修復）
• FlashInfer ≥ 0.6.8 帶 SM120f 編譯（PR #2650）
• CUDA ≥ 12.9

支援 MTP 投機解碼 — 使用 method: qwen3_next_mtp（dense 模型專用，有別於 MoE 的 mtp 方法）。

huihui-ai/Huihui-Qwen3.6-27B-abliterated 的 NVFP4 W4A4 量化版，針對 NVIDIA DGX Spark (GB10 SM121) 最佳化，使用 FlashInfer CUTLASS FP4 kernel。

模型資訊

項目	數值
架構	Dense（27B），64 層 — 16 × (3 × Gated DeltaNet → FFN + 1 × Gated Attention → FFN)
基礎模型	Qwen/Qwen3.6-27B
微調者	huihui-ai（abliteration）
量化者	YuYu1015
模型大小	~28.6 GB（NVFP4 + BF16 MTP/GDN 層，原版 BF16 約 71.9 GB）
Context 長度	最高 262,144 tokens
思考模式	支援（enable_thinking: true/false）
工具呼叫	支援（qwen3_coder parser）
MTP	內建 MTP 權重（qwen3_next_mtp 方法）
DFlash	相容 z-lab/Qwen3.6-27B-DFlash

量化詳情

此模型在 llm-compressor 官方流程上堆疊三項策略（ACD）：

策略	說明
A. 官方基線	Qwen3ForCausalLM + save_mtp_tensors_to_checkpoint（量化後保留 MTP 權重）
C. 混合領域校準	ultrachat_200k（128 對話）+ Nemotron-Post-Training-Dataset-v2（128 推理）共 256
D. 黃金比例參數	num_calibration_samples=256、max_seq_length=4096（品質 > 數量）

B 策略（最後層保護）與 vLLM fused kernel 不相容：對 transformer block 內的 linear 層做 partial ignore 會觸發 ValueError: All projections need to have same quantization scheme but found multiple。

E 策略（SpinQuant R1+R2）與 multi-modal config 不相容：llm-compressor 的 get_head_dim 只讀頂層 config，不讀 Qwen3.6 巢狀的 text_config。

項目	數值
方法	llm-compressor（main）+ compressed-tensors（main）
方案	NVFP4 W4A4（E2M1 + FP8 逐群縮放，群組大小 16）
格式	compressed-tensors
校準資料集	HuggingFaceH4/ultrachat_200k（128）+ nvidia/Nemotron-Post-Training-Dataset-v2（128）
校準樣本總數	256
校準序列長度	4096
量化硬體	NVIDIA DGX Spark（GB10, 128GB 統一記憶體）
環境	transformers>=5.0,<6 + llm-compressor main + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

保留 BF16 的層

層 pattern	原因
re:.*lm_head	輸出頭，對量化雜訊敏感
re:.*embed_tokens$	輸入嵌入
re:visual.* / re:model.visual.*	視覺編碼器
re:.*mlp.gate$	FFN 門控（dense 模型；必須 BF16）
re:.*linear_attn.*	Gated DeltaNet (Mamba) 層 — 量化後可能輸出零
mtp.*（所有 MTP 權重）	量化後透過 save_mtp_tensors_to_checkpoint 以 BF16 重新掛回

投機解碼

本模型支援兩種投機解碼方式：

MTP（內建權重，此 abliterated 變體的建議預設 — 詳見頂部警告）：

--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'

DFlash（獨立 drafter，建議於非 abliterated 版本 / 單用戶 / 低併發使用）：

--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}'

注意：混合 GDN 架構下 MTP 可能觸發 state-rollback bug（vLLM #39273），高 rejection rate 時輸出可能退化。若遇輸出品質下降，請將 num_speculative_tokens 降為 1。

vLLM 部署

vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3.6-27b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --attention-backend flash_attn \
    --kv-cache-dtype auto \
    --gpu-memory-utilization 0.80 \
    --max-model-len 131072 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 8 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --performance-mode throughput \
    --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
    --trust-remote-code \
    --language-model-only

DGX Spark (SM121) 相容性說明

• 原生 W4A4 已確認 透過 FlashInfer CUTLASS NVFP4 kernel
• log 驗證：Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
• FP8 KV cache 與 GDN non-causal attention 不相容，請使用 --kv-cache-dtype auto
• Dense 模型比 35B BF16 更小，128 GB UMA 上可支援更大的 max-model-len
• --language-model-only 跳過視覺編碼器 profiling，加速純文字推理啟動
• UMA 架構啟動前請先清除 page cache：sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

已知限制

• Fused q_proj / k_proj / v_proj 的 per-tensor global scale 可能不一致，vLLM 載入時會印警告。一般精度影響輕微，但在嚴格 tool-calling JSON schema 下可能可測得。
• DFlash drafter 是以原版 Qwen3.6-27B 訓練，非 abliterated 變體 — 接受率可能較原版低。

安全警告

此模型已移除安全過濾機制（abliterated），可能產生敏感、爭議性或不當內容。使用者須自行承擔所有風險與法律責任，並確保使用方式符合當地法規與倫理標準。不適用於公開或生產環境。

致謝

• 原始模型：Qwen/Qwen3.6-27B，Alibaba Qwen 團隊
• 去審查：huihui-ai
• NVFP4 量化：YuYu1015，於 NVIDIA DGX Spark (GB10) 上完成
• 量化工具：llm-compressor，vLLM Project
• 參考量化：sakamakismile/Qwen3.6-27B-NVFP4
• 敏感度分析：Diagnosing FP4 Inference (arXiv 2603.08747)