Qwen3.6-27B-FABQ-RC

Model Description

FABQ-RC (Fisher-Adaptive Binary Quantization with Residual Codebooks) is a 1-bit quantization method for large language models that adapts per layer rather than using a fixed blocksize. This quantization of Qwen3.6-27B achieves ~1.18 bits per parameter while preserving quality through four key innovations.

Property	Value
Base Model	Qwen/Qwen3.6-27B
Quantization Method	FABQ-RC
Format	Custom PyTorch state dict (.pth)
Bits per Parameter	~1.18 bpw
Precision Allocation	Top 5% channels → int8, 95% → binary ±1
Blocksize	Adaptive per-layer (selected from {16, 32, 64, 128, 256})
Residual Codebook	256-entry k-means centroids, per-layer
Architecture	Qwen3.5 hybrid (DeltaNet + Gated Attention), 27B dense, 64 layers, hidden 5120
Modalities	Text only (vision encoder not included)
Context Length	262,144 tokens

How FABQ-RC Works

1. Fisher-Weighted Channel Importance — Per-channel Fisher Information (expected gradient²) determines which channels matter for loss.
2. Mixed-Precision Core — Top 5% channels → int8, rest → binary ±1.
3. Adaptive Blocksize — Per-layer sweep {16, 32, 64, 128, 256}, picked by Fisher-weighted reconstruction error.
4. Residual Codebook — k-means (256 centroids) on quantization residuals corrects systematic binary bias.

Files in This Repo

File	Size	Description
quantized_model.pth	52.9 GB	FABQ-RC quantized state dict (int8 cores + binary weights + codebooks + Fisher masks)
config.json	4.3 kB	Architecture metadata, mirrors Qwen/Qwen3.6-27B text config
generation_config.json	202 B	Sampler defaults
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt	~23 MB	Tokenizer (mirrored from base, unchanged)
chat_template.jinja	7.8 kB	Chat template (mirrored from base)
README.md	this file	Model card

Note: the quantized_model.pth is a custom format. Standard transformers / vLLM / llama.cpp will NOT load it. See Loading below.

Loading & Inference

The loader lives in the FABQ-RC project repo: toxzak/fabq-rc, specifically Main-FABQ-RC-Notebook.ipynb. The notebook contains the FABQ_RC_Model class that reconstructs the binary weights, applies the int8 cores, and runs the residual codebook lookup at inference time.

# Pseudocode — see Main-FABQ-RC-Notebook.ipynb for the real loader
import torch
from fabq_rc import FABQ_RC_Model  # from toxzak/fabq-rc repo

state_dict = torch.load("quantized_model.pth", map_location="cuda")
model = FABQ_RC_Model.from_quantized_state(state_dict).cuda()
output = model.generate("The future of 1-bit quantization is", max_new_tokens=128)

Requirements

• PyTorch with CUDA (custom 1-bit matmul kernel; no native ops exist for the mixed int8/binary layout)
• ~16 GB VRAM minimum (1.18 bpw × 27B params ≈ 4 GB weights + activations)
• transformers >= 4.57.1 for tokenizer/config parsing
• The FABQ-RC loader from the project repo (NOT pip-installable as of this writing)

Why Not GGUF?

The companion bucket (toxzak/Qwen3.6-27B-Q1_K-GGUF-bucket) contains a 3.36 GB GGUF export of the binary weights only, but that export drops the int8 cores and the residual codebook. It is a partial export and will produce degraded output. The .pth in this repo is the complete artifact.

Limitations

• Custom format: No standard runtime can load it without the FABQ-RC loader.
• Weight-only quantization: Activations are not quantized.
• No vision encoder: The vision tower from Qwen3.6-27B is not included; this artifact is text-only.
• Single-domain calibration: C4 only; may not generalize to all domains.
• Not llama.cpp compatible: Despite the bucket name, the .gguf shard in the companion bucket is not a runnable llama.cpp model.

Training / Quantization Details

Property	Value
Method	FABQ-RC (see spec)
Calibration	C4 dataset, 2048 samples
Sequence Length	32 tokens
Hardware	A100 80GB GPU
Method Origin	Designed by Zachary Maronek, 2026-04-05

Benchmark Comparison

Method	bpw	Perplexity	Notes
FP16	16.0	baseline	Qwen3.6-27B full precision
Q1_0_g128	1.125	degraded	Bonsai's format
BiLLM	1.08	~8.41 (70B)	Best prior work
FABQ-RC	~1.18	TBD	This artifact

Related Artifacts

• toxzak/Qwen3.6-27B-Q1_K-GGUF-bucket — partial GGUF export (binary weights only)
• toxzak/fabq-rc — full project repo with loader
• toxzak/Qwen3.6-35B-A3B-FABQ-RC — 35B-A3B variant

Citation

@misc{fabqrc2026,
    author = {Zach Maronek},
    title = {FABQ-RC: Fisher-Adaptive Binary Quantization with Residual Codebooks},
    year = {2026},
    url = {https://huggingface.co/toxzak/fabq-rc}
}

License

Apache 2.0 — same as the base model Qwen/Qwen3.6-27B.

---

FABQ-RC by Zach Maronek · 2026