mratsim/Qwen3.5-397B-A17B-EXL3

Qwen3.5-397B-A17B

Overview

This repo strives to provide the highest quality quant for specific target sizes.

• Targets 192 GiB of VRAM: 2x RTX Pro 6000 or 8x RTX 3090
• Very long-context. It supports 524K context with unquantized FP16 KV-cache or 1M context with 8-bit KV-cache

Highlights of the quant

• EXL3 quantization strategy is state-of-the-art. It uses Hadamard rotations (predating Turboquant by 2 years, including for KV-cache quantization) to spread value spikes on random directions to ease quantization + Trellis / lattice codebooks which have been extremely useful for video compression/quantization (coming from rate-distortion theory).

  • Writeup by EXL3 author (including benches): https://github.com/turboderp-org/exllamav3/blob/v0.0.29/doc/exl3.md
  • https://www.together.ai/blog/even-better-even-faster-quantized-llms-with-qtip

• TabbyAPI (frontend) / ExllamaV3 (backend) support PagedAttention, true continuous batching and ragged concurrent requests like vLLM and SGLang.

  💡 This is unlike llama.cpp / ik-llama.cpp / ollama which use fixed slots of size context / preconfigured_concurrency so for a base 262144 context, with a concurrency of 2, only 131072 token can be used by a single user even if no other query is in flight.

• In theory, tensor parallelism is possible (it is used for my GLM-4.7 quant), however sharding Gated Delta Nets (Qwen3.5 linear attention) is still pending.

• Quantization makes sure that all experts are activated. (The alternative being a large and extremely diverse dataset to ensure comprehensive activation ranges of all experts)

  💡**[Click me!]** Visual showcase of why ensuring quantization of all MoE experts is important

  • Source: https://avtc.github.io/aquarium-side-by-side/
  • Context: https://github.com/ModelCloud/GPTQModel/pull/2235

  

Configuration

Historically TabbyAPI didn't support reasoning and tool-calling, this has been solved as of Apr 12, 2026 commit #32eed618. The quant can be configured with the following config.yaml for proper reasoning and tool-calling.

config.yml (Only what is modified from default)

# Options for model overrides and loading
# Please read the comments to understand how arguments are handled
# between initial and API loads
model:
  # An initial model to load.
  # Make sure the model is located in the model directory!
  # REQUIRED: This must be filled out to load a model on startup.
  model_name: "Qwen3.5-397B-A17B"

  # Max sequence length (default: min(max_position_embeddings, cache_size)).
  # Set to -1 to fetch from the model's config.json
  # max_seq_len: -1
  # max_seq_len: 1048576
  max_seq_len: 524288

  # Size of the key/value cache to allocate, in tokens (default: 4096).
  # Must be a multiple of 256.
  # ExllamaV2 note: On AMD GPUs and NVIDIA GPUs older than Ampere, this value
  # is ignored. Please use max_seq_len
  # cache_size: 1310720
  cache_size: 655360

  # Enable different cache modes for VRAM savings (default: FP16).
  # Possible values for exllamav2: 'FP16', 'Q8', 'Q6', 'Q4'.
  # For exllamav3, specify the pair k_bits,v_bits where k_bits and v_bits are integers from 2-8 (i.e. 8,8).
  cache_mode: FP16   # For 524288 context / 655360 cache
  # cache_mode: 8,8  # For 1M context,
  # cache_mode: 6,5  # For large cache, dequantizing can be costly in terms of tok/s

  # Load model with tensor parallelism.
  # Falls back to autosplit if GPU split isn't provided.
  # This ignores the gpu_split_auto value.
  # tensor_parallel: true
  tensor_parallel: false

  # Reserve VRAM used for autosplit loading (default: 96 MB on GPU 0).
  # Represented as an array of MB per GPU.
  autosplit_reserve: [1024]

  # Regarding rope_scaling, the model defaults to 0.25
  # in config.json so this covers 1M context natively.

  # Chunk size for prompt ingestion (default: 2048).
  # A lower value reduces VRAM usage but decreases ingestion speed.
  # NOTE: Effects vary depending on the model.
  # An ideal value is between 512 and 4096.
  chunk_size: 4096

  # Enables vision support if the model supports it. (default: False)
  vision: true

  # Enable reasoning parser (default: False).
  # Do NOT enable this if the model is not a reasoning model (e.g. deepseek-r1 series)
  reasoning: true

  # Tool format, e.g. 'qwen3_coder'. See docs for supported formats. If left blank,
  # tool calls from the model will not be parsed by the server.
  tool_format: qwen3_coder

# Options for Sampling
sampling:
  # Select a sampler override preset (default: None).
  # Find this in the sampler-overrides folder.
  # This overrides default fallbacks for sampler values that are passed to the API.
  # NOTE: safe_defaults is noob friendly and provides fallbacks for frontends that don't send sampling parameters.
  # Remove this for any advanced usage.
  # override_preset: safe_defaults
  override_preset: Qwen3.5

  # Qwen3.5.yml contains
  # -----------
  # temperature:
  #   override: 0.7
  #   force: false
  # top_p:
  #   override: 0.8
  #   force: false
  # top_k:
  #   override: 20
  #   force: false

The Qwen3.5.yml sampling file has the official recommended settings and is expected in the subdirectory SAMPLERS_PATH="${DIR}/sampler_overrides" from the launch_script

temperature:
  override: 0.7
  force: false
top_p:
  override: 0.8
  force: false
top_k:
  override: 20
  force: false
# min_p:
#   override: 0
#   force: false

Launch script and Dockerfile

Launch script

#!/bin/bash
set -euo pipefail

TABBYAPI_VERSION="latest"
# IMAGE=ghcr.io/theroyallab/tabbyapi:"$TABBYAPI_VERSION"
IMAGE=tabby-202604-cu128

PUBLIC_PORT="5000"
DIR=$(realpath "$(dirname "${BASH_SOURCE[0]}")")
CONFIG_PATH="${DIR}/config.yml"
SAMPLERS_PATH="${DIR}/sampler_overrides"

HF_CACHE=<path/to>/huggingface-hub
LOCAL_MODELS=<path/to>/local_models

mkdir -p "${HF_CACHE}"
mkdir -p "${LOCAL_MODELS}"

podman run --replace --detach --restart on-failure:5 \
  --name="tabbyAPI" \
  --device nvidia.com/gpu=all \
  --security-opt=label=disable \
  --ipc=host \
  -p "${PUBLIC_PORT}:5000/tcp" \
  -v "${LOCAL_MODELS}:/app/models:ro" \
  -v "${CONFIG_PATH}:/app/config.yml:ro" \
  -v "${SAMPLERS_PATH}:/app/sampler_overrides:ro" \
  -v "${HF_CACHE}:/root/.cache/huggingface:rw" \
  $IMAGE

Dockerfile to build TabbyAPI and Exllamav3

My Docker image is generated through: https://github.com/mratsim/llmops/blob/9b9bbfd/exl3/tabbyAPI-exl3-Dockerfile

via the command TMPDIR=$HOME/AI/tmpdir/tabbyAPI-build podman build -t tabby-202604-cu128 -f Dockerfile-tabby-202604

TabbyAPI/Exllamav3 Dockerfile

# Use an official CUDA runtime with Ubuntu as a parent image
FROM nvidia/cuda:12.8.1-runtime-ubuntu24.04

# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    ca-certificates \
    python3.12 \
    python3-pip \
    python3.12-venv \
    python3.12-dev \
    git \
    && rm -rf /var/lib/apt/lists/*

# Create a virtual environment
RUN python3 -m venv /opt/venv

# Activate the venv and set the PATH
ENV PATH="/opt/venv/bin:$PATH"

# Upgrade pip and install uv
RUN pip install --no-cache-dir --upgrade pip

# Set the working directory in the container
WORKDIR /app

# Clone tabbyAPI repository.
RUN git clone https://github.com/theroyallab/tabbyAPI.git /app

# Install packages specified in pyproject.toml cu12, extras
# RUN pip install --no-cache-dir .[cu12,extras]
RUN pip install --no-cache-dir .[cu12]

# Triton needs `apt get install python3.12-dev` for <Python.h>
RUN pip install triton flash-linear-attention

# Impossible to compile by itself, if fails in PyTorch cpp_extension.py
# with a 404 error
# similar to https://github.com/Dao-AILab/causal-conv1d/issues/4
RUN pip install https://github.com/Dao-AILab/causal-conv1d/releases/download/v1.6.1.post4/causal_conv1d-1.6.1+cu12torch2.9cxx11abiTRUE-cp312-cp312-linux_x86_64.whl

# Make port 5000 available to the world outside this container
EXPOSE 5000

# Set the entry point
ENTRYPOINT ["python3"]

# Run main.py when the container launches
CMD ["main.py"]

Acknowledgements

@Goldkoron's own quantization for llama.cpp has been foundational to tune this quant, especially his sensitivity analysis of expert layers:

Besides, @Neurosenko's 3, 4, 5 bpw and @MikeRoz's 2 bpw quant have been used in the mix. And cpral's quant bench table has been reused for presentation.

And of course @turboderp for the amazing library, especially the tooling around quantization and quality measurement.

KL-divergence / quant quality

This has been measured with exllamav3 evals/model_diff which by default (-r 100) evaluates on 100 samples of 2048 tokens from wikitext2 dataset.

I reuse the figures from Neurosenko and cpral's quant for other quants and manually confirmed that their 3bpw and 3.536bpw matches my own manual measurements.

Quant	Size (GiB)	Actual bpw	PPL	KL-div (q→o)	KL-div (o→q)	Top-1	Top-2	Top-3	Top-4	Top-5
MikeRoz 2.0bpw	97	2.00	5.072	0.516	0.821	76.1%	41.3%	18.6%	7.5%	2.9%
MikeRoz 2.08bpw	100	2.08	3.386	0.121	0.163	89.3%	62.6%	38.3%	21.6%	11.7%
NeuroSenko 3.0bpw	142	3.00	3.220	0.0674	0.0776	91.9%	68.4%	44.5%	26.6%	14.8%
NeuroSenko 3.03bpw_opt	143	3.03	3.173	0.0474	0.0531	93.5%	73.4%	51.1%	32.9%	20.1%
mratsim 3.47bpw (this repo)	160	3.47	3.096	0.0203	0.0216	96.0%	82.2%	64.7%	48.1%	34.1 %
cpral 3.536bpw	167	3.54	3.116	0.0286	0.0313	95%	78.7%	59.2%	41.7%	28.0%
NeuroSenko 4.0bpw	188	4.00	3.101	0.0203	0.0210	95.7%	81.0%	62.3%	44.7%	30.5%
NeuroSenko 4.03bpw_opt	189	4.03	3.082	0.0149	0.0153	96.3%	83.9%	67.2%	50.7%	36.6%
NeuroSenko 5.0bpw	234	5.00	3.067	0.0079	0.0079	97.3%	87.6%	73.9%	59.0%	45.3%
original	751	16.00	3.053	—	—	—	—	—	—	—

Recipe

The mixing recipe to produce the quant is available at qwen3.5-override-13.yaml

---

Qwen3.5 Highlights

Qwen3.5 features the following enhancement:

• Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models across reasoning, coding, agents, and visual understanding benchmarks.

• Efficient Hybrid Architecture: Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency and cost overhead.

• Scalable RL Generalization: Reinforcement learning scaled across million-agent environments with progressively complex task distributions for robust real-world adaptability.

• Global Linguistic Coverage: Expanded support to 201 languages and dialects, enabling inclusive, worldwide deployment with nuanced cultural and regional understanding.

• Next-Generation Training Infrastructure: Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent scaffolds and environment orchestration.

For more details, please refer to our blog post Qwen3.5.

Model Overview

• Type: Causal Language Model with Vision Encoder
• Training Stage: Pre-training & Post-training
• Language Model
  • Number of Parameters: 397B in total and 17B activated
  • Hidden Dimension: 4096
  • Token Embedding: 248320 (Padded)
  • Number of Layers: 60
    • Hidden Layout: 15 * (3 * (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE))
  • Gated DeltaNet:
    • Number of Linear Attention Heads: 64 for V and 16 for QK
    • Head Dimension: 128
  • Gated Attention:
    • Number of Attention Heads: 32 for Q and 2 for KV
    • Head Dimension: 256
    • Rotary Position Embedding Dimension: 64
  • Mixture Of Experts
    • Number of Experts: 512
    • Number of Activated Experts: 10 Routed + 1 Shared
    • Expert Intermediate Dimension: 1024
  • LM Output: 248320 (Padded)
  • MTP: trained with multi-steps
• Context Length: 262,144 natively and extensible up to 1,010,000 tokens.

Benchmark Results

Language

	GPT5.2	Claude 4.5 Opus	Gemini-3 Pro	Qwen3-Max-Thinking	K2.5-1T-A32B	Qwen3.5-397B-A17B
Knowledge
MMLU-Pro	87.4	89.5	89.8	85.7	87.1	87.8
MMLU-Redux	95.0	95.6	95.9	92.8	94.5	94.9
SuperGPQA	67.9	70.6	74.0	67.3	69.2	70.4
C-Eval	90.5	92.2	93.4	93.7	94.0	93.0
Instruction Following
IFEval	94.8	90.9	93.5	93.4	93.9	92.6
IFBench	75.4	58.0	70.4	70.9	70.2	76.5
MultiChallenge	57.9	54.2	64.2	63.3	62.7	67.6
Long Context
AA-LCR	72.7	74.0	70.7	68.7	70.0	68.7
LongBench v2	54.5	64.4	68.2	60.6	61.0	63.2
STEM
GPQA	92.4	87.0	91.9	87.4	87.6	88.4
HLE	35.5	30.8	37.5	30.2	30.1	28.7
HLE-Verified¹	43.3	38.8	48	37.6	--	37.6
Reasoning
LiveCodeBench v6	87.7	84.8	90.7	85.9	85.0	83.6
HMMT Feb 25	99.4	92.9	97.3	98.0	95.4	94.8
HMMT Nov 25	100	93.3	93.3	94.7	91.1	92.7
IMOAnswerBench	86.3	84.0	83.3	83.9	81.8	80.9
AIME26	96.7	93.3	90.6	93.3	93.3	91.3
General Agent
BFCL-V4	63.1	77.5	72.5	67.7	68.3	72.9
TAU2-Bench	87.1	91.6	85.4	84.6	77.0	86.7
VITA-Bench	38.2	56.3	51.6	40.9	41.9	49.7
DeepPlanning	44.6	33.9	23.3	28.7	14.5	34.3
Tool Decathlon	43.8	43.5	36.4	18.8	27.8	38.3
MCP-Mark	57.5	42.3	53.9	33.5	29.5	46.1
Search Agent³
HLE w/ tool	45.5	43.4	45.8	49.8	50.2	48.3
BrowseComp	65.8	67.8	59.2	53.9	--/74.9	69.0/78.6
BrowseComp-zh	76.1	62.4	66.8	60.9	--	70.3
WideSearch	76.8	76.4	68.0	57.9	72.7	74.0
Seal-0	45.0	47.7	45.5	46.9	57.4	46.9
Multilingualism
MMMLU	89.5	90.1	90.6	84.4	86.0	88.5
MMLU-ProX	83.7	85.7	87.7	78.5	82.3	84.7
NOVA-63	54.6	56.7	56.7	54.2	56.0	59.1
INCLUDE	87.5	86.2	90.5	82.3	83.3	85.6
Global PIQA	90.9	91.6	93.2	86.0	89.3	89.8
PolyMATH	62.5	79.0	81.6	64.7	43.1	73.3
WMT24++	78.8	79.7	80.7	77.6	77.6	78.9
MAXIFE	88.4	79.2	87.5	84.0	72.8	88.2
Coding Agent
SWE-bench Verified	80.0	80.9	76.2	75.3	76.8	76.4
SWE-bench Multilingual	72.0	77.5	65.0	66.7	73.0	69.3
SecCodeBench	68.7	68.6	62.4	57.5	61.3	68.3
Terminal Bench 2	54.0	59.3	54.2	22.5	50.8	52.5

* HLE-Verified: a verified and revised version of Humanity’s Last Exam (HLE), accompanied by a transparent, component-wise verification protocol and a fine-grained error taxonomy. We open-source the dataset at https://huggingface.co/datasets/skylenage/HLE-Verified.
* TAU2-Bench: we follow the official setup except for the airline domain, where all models are evaluated by applying the fixes proposed in the Claude Opus 4.5 system card.
* MCPMark: GitHub MCP server uses v0.30.3 from api.githubcopilot.com; Playwright tool responses are truncated at 32k tokens.
* Search Agent: most search agents built on our model adopt a simple context-folding strategy(256k): once the cumulative Tool Response length reaches a preset threshold, earlier Tool Responses are pruned from the history to keep the context within limits.
* BrowseComp: we tested two strategies, simple context-folding achieved a score of 69.0, while using the same discard-all strategy as DeepSeek-V3.2 and Kimi K2.5 achieved 78.6.
* WideSearch: we use a 256k context window without any context management.
* MMLU-ProX: we report the averaged accuracy on 29 languages.
* WMT24++: a harder subset of WMT24 after difficulty labeling and rebalancing; we report the averaged scores on 55 languages using XCOMET-XXL.
* MAXIFE: we report the accuracy on English + multilingual original prompts (totally 23 settings).
* Empty cells (--) indicate scores not yet available or not applicable.

Vision Language

	GPT5.2	Claude 4.5 Opus	Gemini-3 Pro	Qwen3-VL-235B-A22B	K2.5-1T-A32B	Qwen3.5-397B-A17B
STEM and Puzzle
MMMU	86.7	80.7	87.2	80.6	84.3	85.0
MMMU-Pro	79.5	70.6	81.0	69.3	78.5	79.0
MathVision	83.0	74.3	86.6	74.6	84.2	88.6
Mathvista(mini)	83.1	80.0	87.9	85.8	90.1	90.3
We-Math	79.0	70.0	86.9	74.8	84.7	87.9
DynaMath	86.8	79.7	85.1	82.8	84.4	86.3
ZEROBench	9	3	10	4	9	12
ZEROBench_sub	33.2	28.4	39.0	28.4	33.5	41.0
BabyVision	34.4	14.2	49.7	22.2	36.5	52.3/43.3
General VQA
RealWorldQA	83.3	77.0	83.3	81.3	81.0	83.9
MMStar	77.1	73.2	83.1	78.7	80.5	83.8
HallusionBench	65.2	64.1	68.6	66.7	69.8	71.4
MMBenchEN-DEV-v1.1	88.2	89.2	93.7	89.7	94.2	93.7
SimpleVQA	55.8	65.7	73.2	61.3	71.2	67.1
Text Recognition and Document Understanding
OmniDocBench1.5	85.7	87.7	88.5	84.5	88.8	90.8
CharXiv(RQ)	82.1	68.5	81.4	66.1	77.5	80.8
MMLongBench-Doc	--	61.9	60.5	56.2	58.5	61.5
CC-OCR	70.3	76.9	79.0	81.5	79.7	82.0
AI2D_TEST	92.2	87.7	94.1	89.2	90.8	93.9
OCRBench	80.7	85.8	90.4	87.5	92.3	93.1
Spatial Intelligence
ERQA	59.8	46.8	70.5	52.5	--	67.5
CountBench	91.9	90.6	97.3	93.7	94.1	97.2
RefCOCO(avg)	--	--	84.1	91.1	87.8	92.3
ODInW13	--	--	46.3	43.2	--	47.0
EmbSpatialBench	81.3	75.7	61.2	84.3	77.4	84.5
RefSpatialBench	--	--	65.5	69.9	--	73.6
LingoQA	68.8	78.8	72.8	66.8	68.2	81.6
V*	75.9	67.0	88.0	85.9	77.0	95.8/91.1
Hypersim	--	--	--	11.0	--	12.5
SUNRGBD	--	--	--	34.9	--	38.3
Nuscene	--	--	--	13.9	--	16.0
Video Understanding
VideoMME(w sub.)	86	77.6	88.4	83.8	87.4	87.5
VideoMME(w/o sub.)	85.8	81.4	87.7	79.0	83.2	83.7
VideoMMMU	85.9	84.4	87.6	80.0	86.6	84.7
MLVU (M-Avg)	85.6	81.7	83.0	83.8	85.0	86.7
MVBench	78.1	67.2	74.1	75.2	73.5	77.6
LVBench	73.7	57.3	76.2	63.6	75.9	75.5
MMVU	80.8	77.3	77.5	71.1	80.4	75.4
Visual Agent
ScreenSpot Pro	--	45.7	72.7	62.0	--	65.6
OSWorld-Verified	38.2	66.3	--	38.1	63.3	62.2
AndroidWorld	--	--	--	63.7	--	66.8
Medical VQA
SLAKE	76.9	76.4	81.3	54.7	81.6	79.9
PMC-VQA	58.9	59.9	62.3	41.2	63.3	64.2
MedXpertQA-MM	73.3	63.6	76.0	47.6	65.3	70.0

* MathVision：our model’s score is evaluated using a fixed prompt, e.g., “Please reason step by step, and put your final answer within \boxed{}.” For other models, we report the higher score between runs with and without the \boxed{} formatting.
* BabyVision: our model’s score is reported with CI (Code Interpreter) enabled; without CI, the result is 43.3.
* V*: our model’s score is reported with CI (Code Interpreter) enabled; without CI, the result is 91.1.
* Empty cells (--) indicate scores not yet available or not applicable.