Kimi-K3-REAPgraded-MLX-mxfp4-q8

Kimi-K3, REAP expert-pruned (64%) and converted to MLX — 451 GB, sized to actually load on a 512 GB Apple Silicon machine.

Base: moonshotai/Kimi-K3 — 2.78T total / 104B active, native-multimodal MoE with Kimi Delta Attention, Attention Residuals and a 1M context.

Why this exists

Unpruned Kimi-K3 does not fit on any Mac. The full model is 1.56 TB; even a 2-bit quant is ~870 GB against a 512 GB ceiling. Fitting 4-bit into 512 GB would need ≤1.38 bits/weight.

REAP (Cerebras) scores each expert by gate x ||expert_output|| over a calibration set and keeps the most salient. This build keeps 326 of 896 experts per layer across 92 MoE layers → 1048B params, 451 GB.

Precision: mxfp4, and it is lossless

K3's routed experts ship from Moonshot as MXFP4 (weight_packed + e8m0 weight_scale, group 32). MLX's native mxfp4 mode uses the identical encoding, so the surviving experts here are a bit-exact byte copy of the source — the only information lost in this repo is the pruning itself, not the quantization.

Requantizing those same weights to affine 4-bit would cost ~9.8% mean relative error and be larger (4.5 vs 4.25 bits/weight), so no affine 4-bit tier is published.

Non-expert tensors (mxfp4 global config; attention, shared experts, latent projections, embeddings) are carried at higher precision.

Measured behaviour

Loaded on a 512 GiB M3 Ultra (65 s, 450 GB peak). Verbatim, greedy, unedited:

prompt: 'def merge_intervals(intervals):\n    """Merge overlapping intervals."""\n'
-->     if not intervals:
        return []
    
    intervals.sort(key=lambda x:

prompt: 'The capital of France is'
-->  Paris."},
+        {"The capital of Germany is Berlin."},
+

prompt: '机器学习的基本原理是'
--> ：机器学习是人工智能领域中的信息技术，通过使用计算机技术和相关技术，让

Samples above are verbatim, greedy-decoded, and unedited — including the failure modes. Degradation from the pruning shows up as drift into list-like or source-file-like continuations rather than answering directly, and at heavier prune ratios as outright repetition loops.

Code completions stay structurally correct across every ratio tested, which matches the calibration data: code experts form a dense, self-similar cluster (57% self-overlap in a top-242 set, versus a 27% chance baseline) and so survive pruning better than more diffuse language capability.

Speed

~2.68 tok/s decoding on a 512 GiB M3 Ultra, via mlx_lm.generate or any standard mlx-lm entry point. Prompt processing is considerably faster; the figure above is generation.

Each decoded token reads roughly 87 GB of weights: 25.8 GB of routed experts plus 60.8 GB of non-expert tensors, which every token touches. Against the M3 Ultra's ~819 GB/s that puts the ceiling near 9.5 tok/s, so this build reaches about 28% of what the memory system permits. It is bandwidth-bound, which is the expected regime for a model of this shape.

Note the non-experts dominate per-token traffic despite being ~2% of parameters: all of them are read every token, while only 16 of 326 experts are.

Pruning buys memory, not speed. Per-token traffic depends on top_k and the non-expert precision, never on how many experts are stored, so the 350 GB 179-expert build and the 451 GB 242-expert build both decode at ~5.5 tok/s. Prune harder to fit a smaller machine, not to go faster.

Correction, 2026-07-28. Earlier revisions of this card reported ~0.15 tok/s and described this build as "not interactive". That number came from an internal benchmark that hand-rolled its own decode loop and so never entered the wired_limit context manager mlx_lm.stream_generate applies automatically. The weights were left unwired, and every decoded token faulted them back from SSD instead of reading RAM -- understating this build by roughly 18x. Users were never affected: the normal mlx-lm paths (generate, stream_generate, the CLI, the server) have always wired correctly, so what you measure is the corrected figure above. Thanks to @pudepiedj for reporting the discrepancy.

Quality expectations — read this

This is an aggressive prune. Top-16 routing over 326 experts is 4.9% density, comparable to a REAP-50 of a 256-expert model. Expect noticeable degradation versus full K3. It is the "fits on one machine" build, not a quality build.

Two things work in its favour that a plain expert cull would not have: K3 keeps 2 shared experts that fire on every token regardless of pruning, and its LatentMoE applies RMSNorm to the combined expert output, which partially self-corrects the magnitude lost when experts are removed.

Calibration — graded two-bank precision

Experts are split across two banks at different bit widths: the 121 most salient per layer at mxfp4 (bit-exact copies of Moonshot's weights) and the remainder at 2-bit. That fits 326 experts per layer instead of 242 at the same footprint — 35% more — because the least salient two thirds cost 2.5 bits each instead of 4.25.

MLX pins one bit width per expert tensor, so this needs a custom TwoBankSwitchGLU (bundled in kimi_k3.py). It exploits the sort SwitchGLU already performs: once routed pairs are sorted by expert index, each bank owns a contiguous slice, so neither bank does the other's work.

Saliency retained rises to 68.4% from 59.1% for a uniform 242-expert build at the same size, and Chinese output is measurably better (the uniform build drifts back into restating the prompt; this one does not).

It costs half the decode speed. Measured end to end against the otherwise comparable single-bank build at the same 450 GB footprint: 2.68 tok/s here versus 5.51 for Kimi-K3-REAP73-MLX-mxfp4-q8 -- a 2.06x penalty, which is essentially the full cost of the run-both-and-select scheme the two-bank design exists to avoid. The overhead is a per-layer host sync to find the data-dependent split point, and K3 has 92 MoE layers serialised down the decode path; what amortises inside a single layer does not amortise across 92. This build moves less memory per token than the single-bank one (part of its experts are 2-bit), so the penalty is entirely machinery, not bandwidth.

Take this build if you want the extra experts and better Chinese and can accept half the speed. Take Kimi-K3-REAP73-MLX-mxfp4-q8 otherwise.

Saliency was measured on a deliberately mixed 12.6 MB corpus — 40% code (multi-language + real Python), 30% English web, 15% Chinese, 15% across ja/ru/ko/de/fr/es/ar. The mix matters: whatever a calibration corpus under-represents gets pruned away silently. An earlier attempt using C4's pooled multilingual config left CJK at 0.03% of the corpus, which would have quietly removed the experts handling Chinese.

Usage

Requires mlx-lm plus the bundled kimi_k3.py loader (the architecture is not upstream yet). On a 512 GB machine you must raise the GPU wired limit first — the default is ~75% of RAM, below this model's footprint:

sudo sysctl iogpu.wired_limit_mb=480000
pip install mlx-lm

python - <<'PY'
import os, shutil, mlx_lm
from huggingface_hub import hf_hub_download
for f in ("kimi_k3.py",):
    dst = os.path.join(os.path.dirname(mlx_lm.__file__), "models", f)
    shutil.copy(hf_hub_download("pipenetwork/Kimi-K3-REAPgraded-MLX-mxfp4-q8", f), dst)
PY

mlx_lm.generate --model pipenetwork/Kimi-K3-REAPgraded-MLX-mxfp4-q8 --max-tokens 256 \
    --prompt "Write a Python function that merges overlapping intervals."

kimi_k3_vision.py and kimi_k3_vl/ ship alongside for the vision tower; the image path needs mlx-vlm and is not exercised by mlx_lm.generate.

Provenance

Converted with PipeNetwork/kimi-k3-mlx: a streaming converter (the model never fits in memory at any stage) and a streaming REAP calibration harness. Weights remain under the Kimi K3 License.