🤗 Models | 🖥️ Github | 📄 Technical Report
## Model Summary **A.X K2 EAGLE3** is an EAGLE3 speculative-decoding draft model for [**A.X K2**](https://huggingface.co/skt/A.X-K2), SK Telecom's 688B-total / 33B-active Mixture-of-Experts foundation model. Paired with A.X K2 at its native 256K context, the draft accelerates decoding by proposing multiple candidate tokens per step that A.X K2 verifies in a single forward pass, without changing what the target model would have generated on its own. In production-traffic serving it delivers up to **1.64× throughput** and **28% lower time-per-output-token**, with the largest gains at low-to-moderate concurrency. This is a **drafter-only** checkpoint: it has no standalone use and is loaded automatically by vLLM alongside A.X K2 as a `--speculative-config` argument. ## Key Features - **EAGLE3 Architecture** Built on [`Eagle3DeepseekV2ForCausalLM`](https://huggingface.co/docs/transformers/main/en/model_doc/deepseek_v2), a single-layer MLA decoder that consumes three auxiliary hidden states from A.X K2's layers 2, 30, and 58 (early / middle / late signal) and combines them through a per-auxiliary-state RMSNorm (`fc_norm`) before the fusion layer. - **256K-Native, RoPE-Matched to A.X K2** The draft's YaRN RoPE parameters (factor 2.0, original context 131,072, θ = 1e6) are tuned to match A.X K2's 256K configuration exactly. It is intended for use with A.X K2 at 256K and has not been validated against other context-length or RoPE configurations. - **Full, Unpruned Vocabulary** `draft_vocab_size` equals A.X K2's full 163,840-token vocabulary, so no target-to-draft ID remapping is needed at the logits boundary. - **Lossless by Construction** As with any EAGLE-style draft, every proposed token is verified by A.X K2 itself; the draft only changes decoding speed, not the distribution A.X K2 samples from. ## Model Details | Property | Value | |----------|-------| | Architecture | Eagle3DeepseekV2 (1-layer MLA decoder) | | Target model | [skt/A.X-K2](https://huggingface.co/skt/A.X-K2) (256K, fused attention gate) | | Hidden size | 7168 | | Auxiliary hidden-state layers | 2, 30, 58 (of A.X K2's 61 layers) | | Combine-layer normalization | Per-auxiliary-state RMSNorm before `fc` (`fc_norm: true`) | | Vocab size | 163,840 (unpruned, matches target) | | Context length | 262,144 tokens (256K), YaRN factor 2.0 | | Checkpoint precision | fp16 | | Checkpoint size | ~5.6 GiB | | Recommended `num_speculative_tokens` | 3 | ## Evaluation Results Measured on production-representative mixed traffic (`k=3`, concurrency 16) against A.X K2 serving without a draft: | Metric | Result | |---|---| | Throughput | **1.64×** | | Time-per-output-token (TPOT) | **−28%** (36.9 → 26.5 ms) | | Mean accepted length | 2.24 tokens/step | Speedup is workload-dependent and concentrated in the memory-bandwidth-bound regime: - **At low-to-moderate concurrency** (roughly 1–4 concurrent sequences), TPOT improves by up to **~2×**. - **At saturated concurrency**, where decoding is already compute-bound, the extra draft/verify forward passes can cost throughput rather than gain it — expect **up to ~10% lower** throughput in that regime. - **Per-domain speedup** varies with how much of each response is decode-bound: Korean **2.44×**, math **1.67×**, science **1.51×**, code **1.13×** (code responses tend to be short, leaving less room for the draft to pay off). ## Usage ### vLLM A.X K2 EAGLE3 is served through the same [SKT-AI/vllm](https://github.com/SKT-AI/vllm) fork used for A.X K2 itself — upstream vLLM **v0.23.0** plus A.X K2 support: ```bash git clone -b axk2-v0.23.0 https://github.com/SKT-AI/vllm.git cd vllm pip install -e . # see the vLLM docs for build prerequisites ``` Add a single `--speculative-config` argument to your existing A.X K2 serving command — everything else stays the same: ```bash vllm serve skt/A.X-K2 --tensor-parallel-size