leonsarmiento/grug-35b-v2-6bit-XL-mlx

grug big brain. grug think fast, talk less, get job done. now grug shrink for apple metal — BaseQuant_XL 6/8-bit recipe. router stay raw bf16, expert muscle get 6-bit gym. vision eye still work.

grug-35b-v2 built by ProCreations on top of Ornith-1.0-35B (Qwen3.5 MoE, 35B total / ~3B active). LoRA on attention + DeltaNet + shared paths only — router and experts untouched. v2.1 added long-hunt data, deep-think, stuck-loop escape. Result: grug think in dense grug-speak inside <think>, answer normal english. Old v1 repetition sickness: extinct.

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

how run

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/grug-35b-v2-6bit-XL-mlx --max-tokens 256 --temperature 0.6 --top-p 0.95 --prompt "Hello"

grug prefer --reasoning-parser deepseek_r1 in server. LM Studio work too — vision mmproj included.

Quantization Strategy

Layers	Bits	Rationale (grug explain)
mlp.gate (router), shared_expert_gate, lm_head, shared_expert	bf16	router brain — grug no touch. mistake here = grug pick wrong muscle.
embed_tokens, self_attn, linear_attn	8-bit	every-token layers — 8-bit near-lossless. grug see fine, grug attend fine.
vision_tower, switch_mlp (routed experts)	6-bit	bulk muscle — 256 expert, only 8 wake per token. redundancy absorb damage.

• Bits per weight: 6.808 · Total size: ~28 GB (6 shards) · Group size: 64

Recommended Inference Parameters

Parameter	Value
temperature	0.6 – 1.0
top_p	0.95
top_k	20
min_p	0.01
repeat_penalty	1.05
reasoning_parser	deepseek_r1

MLX Benchmarks (n=50, instruct mode)

Benchmark	grug-35b-v2 6bit-XL	Ornith-35B 6bit-XL	Qwen3.6-35B 6bit-XL	Gemma-4 26B 6bit-XL
MMLU	80%	64%	64%	76%
MMLU_PRO	66%	66%	64%	82%
ARC_CHALLENGE	90%	92%	90%	90%
HUMANEVAL	84%	78%	78%	98%
MBPP	72%	78%	78%	82%
Average	78.4%	75.6%	74.8%	85.6%

Full comparison against 9 local SOTA MoE models in the Local SOTA for 48GB Macs collection.

grug strong on knowledge (MMLU 80%, +16 over Ornith base) and reasoning (MMLU_PRO tied with Ornith at 66%). Coding trails the Gemma-4 champion as expected — grug is a generalist conversational reasoner, not a coding specialist.

Thinking-Mode MATHQA: Reasoning Efficiency (n=10, seed=42)

Same 10 MATHQA questions, thinking mode, 8192-token budget, temp=1.0. 3-stage protocol: initial run → 5-pattern answer extraction → re-test all failures.

Model	Verified Acc	Reason chars/correct	Time/correct	Total Time
grug-35b-v2 6bit-XL	9/10 (90%)	185	4.2s	42s
Gemma-4 26B 6bit-XL	10/10 (100%)	2,420	39.8s	398s
Qwen3.6-35B 6bit-XL	10/10 (100%)	3,562	32.1s	321s
Laguna-XS 2.1 6bit-XL	8/10 (80%)	5,606	45.5s	489s
Ornith-35B 6bit-XL	10/10 (100%)	7,914	63.8s	638s

grug is the token-efficiency champion — 13× fewer reasoning characters per correct answer than the next best model (Gemma-4). On the hardest question (Q636, probability), grug used 220 chars to reason where Ornith used 5,682 and Gemma used 7,085. grug got it wrong, but the reasoning was dense, not lazy.

Failure Analysis (after re-test)

Question	Initial	Re-test	Diagnosis
Q636	✗ D (expected C)	✗ D	Genuine reasoning error — consistent wrong answer, both runs
Q1083	✗ A (expected C)	✓ C	Sampling variance — fixed on retest
Q2872	✗ [NO MATCH]	✓ E	Parsing failure — grug's ultra-short reasoning sometimes lacks a recognizable answer pattern

The one genuine failure (Q636) is a probability word problem — grug's compressed reasoning style trades depth for speed, occasionally missing subtle constraint interactions. The parsing failure (Q2872) reveals a side-effect of extreme token efficiency: with only ~150 chars of reasoning, the answer format can be ambiguous to regex extraction.

Why grug Wins on Efficiency

ProCreations trained grug to think in dense "grug-speak" inside <think> blocks — eliminating filler, restatement, and self-verification loops. Other 35B MoE models produce 2,000–8,000 reasoning chars per question; grug averages 185. This translates to:

• 15× faster wall time per correct answer (4.2s vs 63.8s for Ornith)
• Context window preservation — less reasoning = more room for the actual conversation
• Energy savings — fewer generated tokens = less compute per answer

The trade-off: 90% accuracy vs 100% for Qwen3.6/Gemma/Ornith. For the 1 question grug genuinely missed, verbose reasoning from the other models would have caught the error.

Source Model

Property	Value
Base	Ornith-1.0-35B (Qwen3.5 MoE)
Parameters	35B total / 3B active
Training	LoRA on attention + DeltaNet + shared paths, merged BF16. v2.1 continued training.
Datasets	grug-35b-v2-train + grug-35b-v2-train-v21

Source benchmarks: HumanEval 84.8, MBPP 88.0, GSM8K 92.0, MATH-500 64.7. Highly token-efficient — 93–97% token reduction vs base model across benchmarks. Zero degenerate loops, 100% think blocks closed.

grug made by ProCreations. Base brain by DeepReinforce. Metal shrink by leonsarmiento.