cloudnathan5/gemma-4-12b-it-MTP-GGUF

Important Update: Unsloth assistant GGUFs

Unsloth has released working Gemma 4 12B instruct assistant models that work with upstream llama.cpp for mtp. Check them out here: https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/tree/main/MTP

Generally you can find them at Unsloth's model pages in the files section where there will be an MTP folder containing the assistant model GGUFs.

Originally this model repo had experimental GGUF quantized versions of the gemma-4-12B-it-assistant from google, but I have since then updated them to be quantizations of Unsloth's MTP drafters which are compatible with the latest llama.cpp out of the box.

Gemma-4-12B-IT-QUANTIZED-MTP GGUFs

This model repo contains various GGUF format quantizations of Unsloth's gemma-4-12b-it-BF16-MTP.gguf. I did not include BF16, F16, or Q8_0 because those are available from Unsloth at that link. All of these quantizations were made using llama-quantize from llama.cpp. These models are just the MTP heads which are meant to be used alongside the base model, not on their own.

Example Usage

#!/usr/bin/env bash

# llama-server from llama.cpp
/path/to/llama.cpp/build/bin/llama-server \
 -m /path/to/gemma-4-12b-it.gguf \ # Base model is some version of Gemma 4 12B it
 -md /path/to/gemma-4-12b-it-Q4_K_M-MTP.gguf \ # Draft model for MTP set to one of these MTP models
 --spec-type draft-mtp \ # Set speculative decoding type to MTP
 --spec-draft-n-max 3 \ # Set the max number of tokens the drafter should generate (you can play around with this)
 -ngld 99 # Keep all draft model layers on GPU (keeps draft generation speedy)