pearsonkyle/gemma4-31b-imatrix-mtp-GGUF

🧊 Google/Gemma-4-31B-it · imatrix · GGUF

imatrix (hybrid)

📦 10.2 · 13.4 · 15.6 · 19.8 GiB IQ2_M · IQ3_M · IQ4_XS · Q5_K_S 🧪 2-bit AWQ · +54% tool-arg accuracy 🏗️ llama.cpp f3e1828 🏅 Agent · 100% patch 👁️ Text + Image · mmproj 772 MB ⚡ MTP drafter · 88% accept @ n=1 🎯 Q5_K_S · text+vision+MTP in 24 GB

🧊 What this is

An aggressively compressed (under 3 bpw) IQ2_M quantization of google/gemma-4-31B-it, calibrated from real coding/tool-use logs. The 2-bit ships an AWQ build (activation-aware scaling) — chosen over plain imatrix by a direct tool-call test: +54% tool-argument accuracy and far more consistent at the same 2.85 bpw, because at 2 bits static KLD/PPL doesn't predict whether the model still fills correct tool arguments (see Why the 2-bit is AWQ). Runs in vanilla llama.cpp / Ollama / LM Studio — no custom runtime, no extra inference cost. Higher-bit IQ3_M (3.76 bpw), IQ4_XS (4.36 bpw), and Q5_K_S (5.55 bpw — highest fidelity, KLD 0.025) builds ship plain hybrid imatrix for users with more VRAM.

👁️ Now with vision (text + image input)

Gemma 4 is natively multimodal. This repo ships the model's vision tower as a separate mmproj-gemma-4-31B-it-Q8_0.gguf (772 MB, SigLIP-style 27-layer encoder, Q8_0 — visually lossless vs F16 at ⅔ the size). Pair it with any of the four quant files via --mmproj and the model can see images — describe screenshots, read diagrams, answer questions about a UI, and so on. The text quant is unchanged; vision adds only the small mmproj. See Usage → Vision below.

🎯 The 24 GB build (Q5_K_S)

The new Q5_K_S build (5.55 bpw, 19.85 GiB) is sized so a single 24 GB GPU can host the full stack at once: the 5-bit text trunk (19.85) + the Q4_K_M MTP drafter (0.36) + the Q8 vision mmproj (0.75) = ~21.0 GiB, leaving ~3.0 GiB for a real KV cache (more with --cache-type-k q8_0 --cache-type-v q8_0). Near-FP16 quality (KLD 0.025), images, and speculative decoding — all on one consumer card, no offload.

📉 ~5.6× smaller10.17 GiB on disk vs 57.2 GiB FP16, at ~2.85 bits/weight.

🤖 Actually agentic47% pass / 100% patch on a 10-instance agentic SWE-rebench holdout (IQ4_XS). IQ2_M still resolved 40% — best of every sub-3-bpw arm tested.

🛠️ Standard GGUFLoads anywhere llama.cpp runs. No patches, kernels, or forks.

📊 Unified benchmark & quality table

Agentic metrics from a SWE-rebench holdout run through the OpenAI Agents SDK (10 instances × 3 reps). Static metrics (PPL / KLD / top-p) measured against FP16 on a held-out eval corpus at ctx=4096. KLD column is median for robustness to per-token tails.

Metric	FP16 (ref)	Q5_K_S	IQ4_XS	IQ3_M	IQ2_M
File	—	Q5_K_S.gguf	IQ4_XS.gguf	IQ3_M.gguf	IQ2_M.gguf
Method	—	imatrix	imatrix	imatrix	AWQ + imatrix
Quality	-	⭐⭐⭐⭐	⭐⭐⭐	⭐⭐	⭐
BPW	16.0	5.55	4.36	3.76	2.85
Size (GiB)	57.20	19.85	15.59	13.43	10.17
🤖 Pass Rate	—	40±8%	47±5%	33±12%	40±8% †
🤖 Patch Rate	—	100%	100%	100%	100% †
🤖 Tool Errors	—	11±2%	10±3%	16±2%	16±1% †
🤖 Mean Tokens	—	663K±111K	575K±70K	483K±75K	558K±94K †
📐 PPL	215.5	256.5	319.4	734.1	1040.0
📐 KLD (med)	0.000	0.025	0.073	0.435	1.804
📐 same_top_p	100.0%	85.5%	78.8%	63.1%	43.9%

† IQ2_M now ships an AWQ build (see § Why the 2-bit is AWQ below). The SWE-rebench figures (†) were measured on the earlier imatrix 2-bit; the AWQ swap is justified by a direct tool-call fidelity comparison (param-acc +54%, far tighter run-to-run variance) at the same 2.85 bpw. AWQ also cuts IQ2_M perplexity nearly in half (1959 → 1040) — it trades a hair of median-KLD for a much more usable 2-bit.

Q5_K_S resolves 40% of the holdout (tying the old IQ2_M, ahead of IQ3_M) at 100% patch and a low 11% tool-error rate; IQ4_XS remains the agentic leader at 47% (gap within run-to-run noise).

📌 Sampling & methodology details

Sampling: temperature=0.25, top_p=0.95, top_k=20, max_tokens=32768, ctx=131072, thinking=false. Run on Apple Silicon (Metal); SWE-rebench linux/amd64 images under emulation, so wall-clock is relative, not absolute.

Pass Rate = gold tests pass after agent's patch (real resolution). Patch Rate = non-empty diff produced.

---

🧪 Why the 2-bit is AWQ

Static KLD/PPL measures how closely a quant's logits track FP16 — it does not measure whether the model still fills correct tool-call arguments, which is what agentic use actually needs. At 2 bits the plain-imatrix build keeps decent tool selection but loses argument precision. We replay 25 held-out real tool-use sessions (disjoint from calibration) through llama-server, scoring per assistant turn whether the model picks the right tool (tool-sel) and fills the right arguments (param-acc). 3 reps, mean ± σ, same sampling as above.

Both builds are measured on the same held-out eval + FP16 baseline (static) and the same 25-session replay (agentic). Static: ↓ PPL / ↓ KLD / ↑ top_p is better. Agentic: ↑ is better.

IQ2_M (2.85 bpw)	PPL	KLD (med)	top_p	tool-sel	param-acc	schema-valid
imatrix	1958.7	1.571	46.6%	0.454 ± .071	0.171 ± .082	0.805
AWQ (shipped)	1040.0	1.804	43.9%	0.492 ± .013	0.263 ± .009	0.823

Read the split: on the metric the static table headlines (median-KLD) imatrix looks better (1.571 vs 1.804), and it edges top_p (46.6% vs 43.9%) — yet the AWQ build is the one that actually tool-calls. AWQ activation-aware scaling wins where it counts: +54% tool-argument accuracy (0.17 → 0.26) and — just as important for agents — it collapses run-to-run variance (param-acc ±.009 vs ±.082; the imatrix build swings 0.51/0.48/0.38 on tool-sel across seeds, AWQ holds 0.50/0.50/0.48). (AWQ also nearly halves PPL, 1959 → 1040 — PPL and KLD disagree here because the fold shifts the whole distribution, lowering average surprise while nudging the very top token slightly.) The upshot: the static table alone would have picked the wrong build. All 60 gemma layers are standard attention, so AWQ scales the full network (120 groups); the per-channel scales fold into RMSNorm with no inference cost. The higher-bit builds (IQ3_M+) don't need it — at 3-4 bits the imatrix build already preserves arguments and AWQ's fold slightly perturbs general text, so those ship plain imatrix.

Harness: quant-tuner run_toolcall_reps.py. This lighter replay is the proxy used to rank the 2-bit builds; a full SWE-rebench re-run on the AWQ build is pending (the table's IQ2_M SWE-rebench figures are the prior imatrix build's).

---

🔬 How it was made

• Hybrid imatrix — activation energy E[a²] mixed with weight-column energy ‖W[:,i]‖²·E[a²] per tensor, collected over real coding/tool-use logs + wiki.test.raw via quant-tuner. Used for the IQ3_M / IQ4_XS / Q5_K_S builds.
• AWQ for the 2-bit — the IQ2_M build additionally applies activation-aware weight scaling: each input channel is rescaled by mean(|x|)^α (α grid-searched on the tool-use logs, sampled across the whole corpus) and the inverse folds into the preceding RMSNorm — mathematically identity in FP16, but it moves the channels that carry tool-argument signal onto finer quant levels. All 60 layers are standard attention, so the fold covers the full network; the imatrix is then collected on the folded weights. Net: +54% tool-argument accuracy vs plain 2-bit imatrix, at no inference cost. See § Why the 2-bit is AWQ.
• IQ2_M codebook — 2-bit E8-lattice non-uniform codes with per-tensor tier bumps (attention output, early ffn_down get more bits). llama-quantize decides the mix.
• Vision mmproj — the model's SigLIP-style vision tower (27 layers, 280 soft tokens/image) exported separately at Q8_0 with convert_hf_to_gguf.py --mmproj (visually lossless, 772 MB), so the encoder stays high-precision while the text path runs at 2 bits. No audio encoder is shipped (the source has none).
• Disjoint splits — calibration (imatrix), validation (per-tensor α gate), and eval (PPL/KLD) come from different corpora; the SWE-rebench holdout never appears in any calibration set.
• Toolchain: quant-tuner for imatrix calibration, llama.cpp @ f3e1828 for final quantization. Calibration logs mined with LogMiner.

---

🚀 Usage

Ollama

ollama run hf.co/pearsonkyle/gemma4-31b-imatrix-mtp-GGUF:IQ2_M

llama.cpp (GPU)

# Build with CUDA (-DGGML_CUDA=OFF for CPU/Metal)
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp/

# Run the server
./llama-server \
    --model gemma-4-31B-it-IQ2_M.gguf \
    --ctx-size 16384 --n-gpu-layers 999 --split-mode layer \
    --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
    --parallel 1 --batch-size 2048 --ubatch-size 512 \
    --host 0.0.0.0 --port 1234

OpenAI-compatible API (Python)

import json, urllib.request

def ask(content, max_tokens=256):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # Gemma 4 is a thinking model — disable or raise max_tokens
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request(
        "http://127.0.0.1:1234/v1/chat/completions",
        json.dumps(body).encode(),
        {"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("What is 1+1?"))

🖼️ Vision (text + image)

Gemma 4 is natively multimodal. The vision tower ships separately as mmproj-gemma-4-31B-it-Q8_0.gguf (772 MB) so you only download it if you need images. It pairs with any of the four quant files (IQ2_M / IQ3_M / IQ4_XS / Q5_K_S) — the text weights are identical; the mmproj just adds the SigLIP encoder + projector.

One-shot from the CLI (llama-mtmd-cli):

./llama-mtmd-cli \
    --model gemma-4-31B-it-IQ4_XS.gguf \
    --mmproj mmproj-gemma-4-31B-it-Q8_0.gguf \
    --image screenshot.png \
    --jinja -ngl 999 --temp 0.2 -n 256 \
    -p "Describe this image. What's in it?"

--jinja is required — Gemma 4's chat template is Jinja-based and the CLI aborts without it. --image can be repeated for multi-image prompts; URLs work too.

⚠️ Thinking + the CLI. Gemma 4 is a reasoning model. From llama-mtmd-cli, leave thinking on and give it enough budget (-n 800+) so the answer survives the reasoning preamble — the --chat-template-kwargs '{"enable_thinking":false}' flag currently returns an empty completion on the CLI path. To get a clean, reasoning-free answer, disable thinking over the HTTP server instead (below).

Vision server — host the quant with the mmproj attached (this is exactly how the worked example above was generated). --jinja is required; the vision tower is loaded via --mmproj:

./llama-server \
    -m gemma-4-31B-it-IQ4_XS.gguf \
    --mmproj mmproj-gemma-4-31B-it-Q8_0.gguf \
    --jinja --ctx-size 8192 --n-gpu-layers 999 \
    --host 127.0.0.1 --port 1234

Vision is purely additive — drop the --mmproj flag and you're back to the identical text-only model.

The OpenAI-compatible /v1/chat/completions endpoint then accepts image_url content parts. With chat_template_kwargs.enable_thinking=false the server returns just the answer (no reasoning preamble). This is the exact call used to generate the mecha prompt above:

import base64, json, urllib.request

with open("mecha.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

body = {
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": (
            "Look at this image and write a single, detailed text-to-image "
            "generation prompt that would recreate it. Cover the subject, colors, "
            "pose, lighting, style, and background. Respond with only the prompt."
        )},
        {"type": "image_url",
         "image_url": {"url": f"data:image/png;base64,{b64}"}},
    ]}],
    "max_tokens": 400,
    "temperature": 0.3,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request(
    "http://127.0.0.1:1234/v1/chat/completions",
    json.dumps(body).encode(),
    {"Content-Type": "application/json"},
)
print(json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"])

Worked example — same image, all three quants

Each shipped quant was handed the mecha on the right and asked to "write a single, detailed text-to-image generation prompt that would recreate it".

Quant	BPW	Generated text-to-image prompt
IQ4_XS	4.36	A high-detail anime-style illustration of a heavy armored mecha robot standing in a powerful, wide-legged stance against a solid black background. The mecha features bulky, angular white and light-grey plating with deep purple accents and glowing neon green energy vents and eyes. In its right hand, it holds a massive, oversized purple railgun, and in its left hand, a glowing neon green energy blade that casts a vibrant light across the armor. The lighting is dramatic and cinematic, with strong rim lighting and high contrast, emphasizing the metallic textures and the bioluminescent glow of the green components. The art style is clean, modern mecha anime with sharp line work and smooth digital shading.
IQ3_M	3.76	Full-body shot of a heavily armored futuristic mecha robot in a powerful standing pose, designed in a detailed 90s retro anime style. The mecha features a bulky, angular white and light-grey armor plating with deep purple accents and glowing neon green energy vents and sensors integrated into the chest, legs, and eyes. It is armed with a massive, oversized purple sniper rifle held in its right hand and a glowing neon green energy sword held in its left. The lighting is dramatic and moody, with a strong rim light highlighting the silhouette against a solid black background. High contrast, clean cel-shaded lines, sharp mechanical detailing, cinematic composition.
IQ2_M	2.85	Full-body shot of a futuristic mecha robot in a stylized anime aesthetic, featuring a heavy armored chassis in a palette of white, grey, and deep purple. The mecha is posed in a powerful stance, holding a large purple futuristic firearm in its right hand and a glowing neon-green energy blade in its left hand. The design includes glowing mint-green accents and circuitry lines across the chest, legs, and head. The lighting is dramatic and moody, with a strong rim lighting and a dark, atmospheric background with subtle purple gradients and a slight digital scanline texture. High-contrast cel-shaded style with clean lines and sharp metallic reflections.

⚡ Speculative decoding (MTP drafter)

This repo also bundles a multi-token-prediction (MTP) drafter at the repo root, mtp-gemma-4-31B-it.gguf (358 MB, Q4_K_M) — a self-quantized conversion of google/gemma-4-31B-it-assistant (arch gemma4-assistant, nextn_predict_layers = 4). The acceptance rates where the model's drafted tokens were accepted by the trunk were measured using three reps over 5 mixed coding/reasoning prompts × 200 tokens, temperature=0.3, thinking off; distinct seeds per rep.

Quant	n=1	n=2	n=3	n=4
Q5_K_S	87.9%	81.8%	73.0%	66.0%
IQ4_XS	86.5%	80.2%	68.6%	64.0%
IQ3_M	87.2%	79.1%	70.8%	64.6%
IQ2_M	83.1%	77.1%	70.6%	61.4%

Acceptance holds up across all four trunks — the highest-fidelity Q5_K_S leads at every draft depth (87.9% at n=1, still 66.0% at n=4), and even the 2-bit IQ2_M accepts 83% of single-token drafts.

How small can the drafter be? Fixing the trunk at IQ2_M and quantizing the drafter itself shows acceptance barely moves down to Q4_K_M, then erodes at 2-bit — and the erosion grows with draft depth (n):

Drafter	size	n=1	n=2	n=3	n=4
Q8_0	514 MB	84.5 ±0.5%	76.7 ±2.5%	69.3 ±1.5%	61.2 ±2.0%
Q6_K	401 MB	84.8 ±0.7%	76.6 ±2.4%	69.2 ±1.4%	61.4 ±1.7%
Q4_K_M (shipped)	358 MB	85.0 ±0.4%	76.8 ±1.9%	69.0 ±0.4%	60.8 ±0.9%
IQ3_M	328 MB	83.9 ±0.9%	75.1 ±1.4%	67.6 ±0.3%	60.6 ±0.7%
IQ2_M	269 MB	83.7 ±0.7%	74.1 ±2.0%	65.9 ±2.3%	57.9 ±2.4%

Q6_K and Q4_K_M are statistically free — indistinguishable from the Q8_0 drafter at every depth (within ±1σ); the drafter only has to be directionally right for the trunk to accept.

Usage — add --model-draft + --spec-type draft-mtp to the server command:

./llama-server \
    -m gemma-4-31B-it-IQ4_XS.gguf \
    --model-draft mtp-gemma-4-31B-it.gguf \
    --spec-type draft-mtp --spec-draft-n-max 4 \
    --jinja -ngl 999 -fa on \
    --host 127.0.0.1 --port 1234

The drafter lives at the repo root so --spec-type draft-mtp auto-discovers it when you load the trunk with -hf (no manual --model-draft needed): llama-server -hf pearsonkyle/gemma4-31b-imatrix-mtp-GGUF:IQ4_XS --spec-type draft-mtp --spec-draft-n-max 4.

---

🪪 License & attribution

• Inherits the Gemma Terms of Use from the base model.
• Base weights: google/gemma-4-31B-it.
• MTP drafter converted from google/gemma-4-31B-it-assistant (same Gemma Terms of Use).
• Calibration + quantization: Quant-Tuner with vendored llama.cpp @ f3e1828.
• Calibration logs mined with LogMiner.