Upload 7 files (#2)

- Upload 7 files (6ef27188e0365f3f48773f35708186054845e243)

.gitattributes

@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+accuracy_plot.png filter=lfs diff=lfs merge=lfs -text
+agentic_coding_benchmarks.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -15,7 +15,6 @@ tags:
 - nvidia
 - pytorch
 - nemotron-3.5
-- latent-moe
 track_downloads: true
 ---
 
@@ -36,59 +35,24 @@ track_downloads: true
   </a>
 </div>
 
+![](./accuracy_plot.png)
+
 ## Model Summary
 
 | | |
 |:---|:---|
 | **Total Parameters** | 30B (3B active) |
-| **Architecture** | LatentMoE — Mamba-2 + MoE + Attention hybrid |
+| **Architecture** | MoE - Mamba-2 + MoE + Attention hybrid |
 | **Context Length** | Up to 1M tokens |
-| **Single-GPU Deployment** | 1× DGX Spark (GB10) or 1× H100; 1× RTX 5090 via llama.cpp (GGUF) |
+| **Single-GPU Deployment** | 1× DGX Spark (GB10) or 1× H100 |
 | **Supported Hardware** | NVIDIA Blackwell (DGX Spark / GB10, GB200, GeForce RTX 5090); NVIDIA Hopper (H100, H200); NVIDIA Ampere via W4A16 |
 | **Supported Languages** | English (and coding languages), Spanish, French, German, Italian, Japanese |
-| **Speculative Decoding** | DFlash for Low Concurrency Data Centre Deployments, and DSpark for DGX Spark Workflows — [Read more below](#speculative-decoding-strategies) |
+| **Speculative Decoding** | DSpark for low-concurrency Data Centre and DGX Spark Workflows — [Read more below](#speculative-decoding-strategies), also provided are MTP (Multi-Token Prediction) and DFlash |
 | **Recommended Sampling** | Temperature 1.0, Top_P 0.95 |
 | **Best For** | Long-running autonomous agents, sub-agent workhorse deployments, and efficient local inference on personal hardware |
 | **License** | [OpenMDW License Agreement, version 1.1](https://raw.githubusercontent.com/OpenMDW/OpenMDW/refs/heads/main/1.1/LICENSE.OpenMDW-1.1) |
 | **Release Date** | August 11, 2026 |
 
-## Quick Start
-
-To get quickly started on DGX Spark (GB10) you can use the following command.
-
-Grab the model:
-
-```shell
-export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
-export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
-```
- 
-Run it with vLLM — this recipe uses DSpark speculative decoding, tuned for DGX Spark. (Nightly: `vllm/vllm-openai:nightly-821717118fc26667dd474b9b0ab81d29259dfc5c`)
- 
-```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --moe-backend marlin \
-    --kv-cache-dtype fp8 \
-    --trust-remote-code \
-    --max-model-len 1048576 \
-    --max-num-batched-tokens 16384 \
-    --enable-prefix-caching \
-    --quantization modelopt_fp4 \
-    --compilation_config.cudagraph_capture_sizes '[1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 1024, 2048, 4096, 8192]' \
-    --mamba-backend flashinfer \
-    --mamba-ssm-cache-dtype float16 \
-    --enable-mamba-cache-stochastic-rounding \
-    --mamba-cache-philox-rounds 5 \
-    --mamba-cache-mode align \
-    --speculative_config.method dspark \
-    --speculative_config.model $DSPARK_CKPT \
-    --speculative_config.num_speculative_tokens 3 \
-    --speculative_config.max_model_len 1048576
-```
-
-For more details on how to deploy and use the model — see the [Quick Start Guide](#quick-start-guide) below!
-
 ## Model Overview
 
 **Model Developer:** NVIDIA Corporation
@@ -108,23 +72,80 @@ NVIDIA Nemotron™ is a family of open models with open weights, training data,
 
 **NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4** is a large language model (LLM) trained by NVIDIA.
 
-The model employs a hybrid **Latent Mixture-of-Experts (LatentMoE)** architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside a number of speculative decoding methods for faster text generation. The model has **3B active parameters** and **30B parameters in total**.
+The model employs a hybrid **Mixture-of-Experts** architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside a number of speculative decoding methods for faster text generation. The model has **3B active parameters** and **30B parameters in total**.
 
 This model is ready for commercial use.
 
+## Quick Start
+
+To get quickly started on DGX Spark (GB10) you can use the following command.
+
+Grab the model:
+
+```shell
+export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
+export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
+```
+ 
+Run it with vLLM — this recipe uses DSpark speculative decoding, tuned for DGX Spark. (vLLM Nightly: `vllm/vllm-openai:v0.27.1`)
+ 
+```shell
+vllm serve --model $MODEL_CKPT \
+  --moe-backend marlin \
+  --kv-cache-dtype fp8 \
+  --enable-prefix-caching \
+  --speculative_config.num_speculative_tokens 3 \
+  --mamba-backend flashinfer \
+  --mamba-cache-mode align \
+  --reasoning-parser nemotron_v3 \
+  --speculative_config.model $DSPARK_CKPT \
+  --tool-call-parser qwen3_coder \
+  --enable-auto-tool-choice
+```
+
+For more details on how to deploy and use the model — see the [Quick Start Guide](#quick-start-guide) below!
+
 ## License/Terms of Use
 
 **Governing Download Terms:** Use of this model is governed by the [OpenMDW-1.1 model license](https://openmdw.ai/license/1-1/).
 
-<!-- ## Computational Load (Internal Only — remove before publishing)
-Cumulative Compute: 6.12 × 10²³ FLOP (BF16, 6ND method). 
+## Benchmarks
 
-Estimated Energy and Emissions for Model Training: ≈ 3.37 × 10⁵ kWh (~337 MWh); ≈ 138 tCO₂e.
--->
+### Reasoning Benchmark Evaluations
 
-## Benchmarks
+We evaluated our model on the following benchmarks:
 
-{{<!-- INSERT: GA benchmark table-->}}
+| Task | Nemotron-3.5-Lightning-30B-A3B-BF16 | Nemotron-3.5-Lightning-30B-A3B-NVFP4 |
+| --- | --- | --- |
+| **General Knowledge** | | |
+| MMLU Pro | 81.94 | 81.62 |
+| AA-Omniscience | 17.50 | 16.63 |
+| **Reasoning** | | |
+| GPQA Diamond (no tools) | 75.44 | 75.57 |
+| HLE (text-only, no tools) | 11.72 | 10.47 |
+| SciCode | 32.60 | 31.38 |
+| **Coding & Agentic** | | |
+| SWE-bench Verified | 51.56 | 52.80 |
+| SWE-bench Multilingual | 39.33 | 36.47 |
+| Terminal-Bench 2.1 | 24.58 | 23.46 |
+| PinchBench | 85.37 | 83.43 |
+| BrowseComp | 36.97 | 36.81 |
+| τ³-bench (Banking) | 9.28 | 9.48 |
+| GDPval-AA-V2 | 832 | 865 |
+| **Instruction Following** | | |
+| IFBench (loose) | 71.88 | 72.88 |
+| **Long Context** | | |
+| AA-LCR | 52.00 | 49.19 |
+
+For reproducibility, the evaluation recipes, installation instructions, and commands for NVIDIA Nemotron 3.5 Lightning were collected and published in [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym/tree/main/nemotron_recipes/lightning-3.5/reproducibility.md). The reported results cover the release evaluation suite, including knowledge and reasoning, instruction following, coding, agentic, tool-use, and long-context. Most evaluations use NeMo Gym-native harnesses while a small subset, including SWE-Bench and Terminal-Bench, used [NeMo Evaluator](https://github.com/NVIDIA-NeMo/Evaluator) natively. The published recipes specify the benchmark-specific containers, prompts, inference parameters, parser configurations, and scoring settings used to produce the results. 
+
+> These numbers were measured with and apply to the official [NVFP4 checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4)
+
+#### Agentic Coding Benchmarks
+
+Additional harness-level coding-agent results for SWE-Bench Verified and Terminal-Bench 2.1 are shown below.
+
+![Agentic Coding Benchmarks](./agentic_coding_benchmarks.png)
 
 ### Deployment Geography: Global
 
@@ -134,7 +155,7 @@ NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 is a general purpose reasoning and c
 
 ### Release Date
 
-Hugging Face — 08/11/2026 {{<!-- add HF collection link-->}}
+[Hugging Face](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4) — 08/11/2026
 
 ## Model Architecture
 
@@ -144,7 +165,7 @@ Hugging Face — 08/11/2026 {{<!-- add HF collection link-->}}
 
 ## Model Design
 
-The model was pre-trained with over 20T tokens and supports up to 1M context length. The pre-training phase used an NVFP4 recipe. It utilizes the **LatentMoE** architecture, where tokens are projected into a smaller latent dimension for expert routing and computation, improving accuracy per byte. The model includes **Multi-Token Prediction (MTP)** layers, which predict multiple future tokens to provide richer training signals.
+The model was pre-trained with over 20T tokens and supports up to 1M context length. The pre-training phase used an NVFP4 recipe. The model includes **Multi-Token Prediction (MTP)** layers, which predict multiple future tokens to provide richer training signals.
 
 ## Training Methodology
 
@@ -208,102 +229,86 @@ All deployment snippets below assume:
 export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
 ```
  
-And for DSpark and DFlash:
+And for DSpark:
  
 ```shell
 export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
 ```
  
-```shell
-export DFLASH_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash
-```
- 
 ### **Speculative Decoding Strategies**
  
-Lightning 3.5 ships with two external draft models for speculative decoding — pick the one that matches your deployment, or serve without speculative decoding for maximum-throughput batch workloads:
- 
-* **DFlash:** An external draft-model checkpoint tuned for **low concurrency data centre deployments**, minimizing per-request latency when batch sizes are small. The 1× H100 vLLM recipe drafts 3 speculative tokens per step (the GB200 recipe drafts 5); llama.cpp supports up to 7 via `--spec-draft-n-max`.
-* **DSpark:** A draft-model configuration tuned specifically for **DGX Spark (GB10) workflows**, pairing draft settings with the Marlin MoE backend and the memory footprint of a single DGX Spark unit.
+Lightning 3.5 ships with two external draft models for speculative decoding as well as MTP (Multi-Token Prediction). While we currently recommend DSpark for all cases - your usecase may align with DFlash and MTP:
  
-| Strategy | Best for | Tested on |
-|:---|:---|:---|
-| **None** | Maximum-throughput, high-concurrency batch serving | 1× H100, 8× H100 (vLLM); 1× H100 (TRT-LLM, SGLang) |
-| **DFlash** | Latency-sensitive, low-concurrency serving | 1× H100, 1× GB200 (vLLM); 1× RTX 5090 (llama.cpp) |
-| **DSpark** | Interactive / local DGX Spark workflows | 1× DGX Spark (vLLM) |
- 
-The snippets in each backend section below show how to launch with the corresponding strategy.
+* **DSpark:** A semi-autoregressive speculative-decoding drafter that proposes a whole block of candidate tokens in a single forward pass from a parallel backbone. This is recommended for DGX Spark, as well as low-concurrency data centre deployments.
+* **DFlash:** A speculative-decoding drafter that uses a lightweight block-diffusion model to generate an entire draft block in one forward pass.
+* **MTP:** A modeling technique that trains the network to predict several future tokens at each position instead of only the next one.
  
 
-### **vLLM**
+### **vLLM** 
+
+> For more indepth instructions on how to deploy through vLLM, head [here](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/usage-cookbook/Nemotron-3.5-Lightning/vllm_cookbook.ipynb)
  
-* Nightly container: `vllm/vllm-openai:nightly-821717118fc26667dd474b9b0ab81d29259dfc5c`
-> All vLLM snippets below serve on the default port `8000`, matching the [API Client](#api-client) examples.
+* vLLM Nightly: `vllm/vllm-openai:v0.27.1`
  
 #### **1x DGX Spark (GB10)**
  
 **Specdec method - DSpark:**
  
 ```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --moe-backend marlin \
-    --kv-cache-dtype fp8 \
-    --trust-remote-code \
-    --max-model-len 1048576 \
-    --max-num-batched-tokens 16384 \
-    --enable-prefix-caching \
-    --quantization modelopt_fp4 \
-    --compilation_config.cudagraph_capture_sizes '[1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 1024, 2048, 4096, 8192]' \
-    --mamba-backend flashinfer \
-    --mamba-ssm-cache-dtype float16 \
-    --enable-mamba-cache-stochastic-rounding \
-    --mamba-cache-philox-rounds 5 \
-    --mamba-cache-mode align \
-    --speculative_config.method dspark \
-    --speculative_config.model $DSPARK_CKPT \
-    --speculative_config.num_speculative_tokens 3 \
-    --speculative_config.max_model_len 1048576
+vllm serve --model $MODEL_CKPT \
+  --moe-backend marlin \
+  --kv-cache-dtype fp8 \
+  --max-model-len 1048576 \
+  --enable-prefix-caching \
+  --speculative_config.num_speculative_tokens 3 \
+  --mamba-backend flashinfer \
+  --mamba-cache-mode align \
+  --reasoning-parser nemotron_v3 \
+  --speculative_config.method dspark \
+  --tool-call-parser qwen3_coder \
+  --enable-auto-tool-choice
 ```
  
- 
 #### **1x H100**
  
-For high concurrency deployments, use the following:
+For max throughput deployments, use the following configuration, no speculative decoding strategy is best for this serving configuration, and due to memory constraints the Mamba cache `dtype` is set as FP16:
  
 ```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_USE_FASTOKENS=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --max-num-seqs 512 \
-    --trust-remote-code \
-    --max-model-len 1048576 \
+vllm serve --model $MODEL_CKPT \
+    --max-num-seqs 256 \
     --max-num-batched-tokens 16384 \
     --enable-prefix-caching \
     --async-scheduling \
     --mamba-backend flashinfer \
+    --moe-backend humming \
+    --linear-backend humming \
+    --mamba-ssu-algorithm horizontal \
+    --mamba-cache-mode align \
     --mamba-ssm-cache-dtype float16 \
     --enable-mamba-cache-stochastic-rounding \
     --mamba-cache-philox-rounds 5 \
-    --mamba-cache-mode align
+    --reasoning-parser nemotron_v3 \
+    --tool-call-parser qwen3_coder \
+    --enable-auto-tool-choice
 ```
  
-For low-concurrency deployments, you can use this following command, leveraging DFlash:
+For interactive usage scenarios (achieving 40+ TPS/User) use a lower concurrency (<=128) with DSpark:
  
 ```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_USE_FASTOKENS=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --trust-remote-code \
-    --max-num-seqs 512 \
-    --max-model-len 1048576 \
-    --max-num-batched-tokens 8192 \
+vllm serve --model $MODEL_CKPT \
+    --max-num-seqs 128 \
     --enable-prefix-caching \
     --async-scheduling \
-    --speculative_config.method dflash \
-    --speculative_config.model $DFLASH_CKPT \
+    --speculative_config.model $DSPARK_CKPT \
     --speculative_config.num_speculative_tokens 3 \
+    --mamba-ssu-algorithm horizontal \
     --mamba-backend flashinfer \
     --mamba-ssm-cache-dtype float16 \
     --enable-mamba-cache-stochastic-rounding \
-    --mamba-cache-philox-rounds 5
+    --mamba-cache-philox-rounds 5 \
+    --reasoning-parser nemotron_v3 \
+    --tool-call-parser qwen3_coder \
+    --enable-auto-tool-choice
 ```
  
 #### **8x H100**
@@ -311,270 +316,135 @@ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_USE_FASTOKENS=1 vllm serve --model $MODEL_C
 For long-context, multi-GPU serving (TP8 with expert parallelism):
  
 ```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_USE_FASTOKENS=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --moe-backend flashinfer_cutlass \
-    --trust-remote-code \
-    --max-num-batched-tokens 4096 \
+vllm serve --model $MODEL_CKPT \
     --mamba-backend flashinfer \
+    --async-scheduling \
     --enable-prefix-caching \
     --mamba-cache-mode align \
-    --max-model-len 1048576 \
     --enable-expert-parallel \
-    --tensor-parallel-size 8
+    --tensor-parallel-size 8 \
+    --reasoning-parser nemotron_v3 \
+    --tool-call-parser qwen3_coder \
+    --enable-auto-tool-choice
 ```
  
 #### **1x GB200**
  
-**Specdec method - DFlash:**
- 
 ```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_USE_FASTOKENS=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --max-num-seqs 64 \
-    --max-model-len 1048576 \
+vllm serve --model $MODEL_CKPT \
     --max-num-batched-tokens 10240 \
     --no-enable-prefix-caching \
     --async-scheduling \
-    --speculative_config.method dflash \
-    --speculative_config.model $DFLASH_CKPT \
+    --speculative_config.model $DSPARK_CKPT \
     --speculative_config.num_speculative_tokens 5 \
-    --mamba-backend flashinfer
+    --mamba-backend flashinfer \
+    --reasoning-parser nemotron_v3 \
+    --tool-call-parser qwen3_coder \
+    --enable-auto-tool-choice
 ```
  
-#### **W4A16 — Hopper / Ampere**
- 
-The same checkpoint also serves via W4A16 kernels, extending coverage to Ampere-class GPUs. 
+#### **W4A16 — Ampere**
  
-For maximum-throughput, high-concurrency serving:
+The same checkpoint also serves via W4A16 kernels, extending coverage to Ampere-class GPUs:
  
 ```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
+vllm serve --model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
     --moe-backend humming \
-    --max-num-seqs 512 \
-    --trust-remote-code \
-    --max-model-len 1048576 \
+    --linear-backend humming \
+    --max-num-seqs 256 \
     --max-num-batched-tokens 32768 \
     --enable-prefix-caching \
     --async-scheduling \
     --quantization modelopt_fp4 \
     --mamba-backend flashinfer \
-    --mamba-ssm-cache-dtype float16 \
-    --enable-mamba-cache-stochastic-rounding \
-    --mamba-cache-philox-rounds 5 \
     --mamba-cache-mode align \
-    --mamba-ssu-algorithm horizontal
-```
- 
-For low-concurrency deployments, you can use this following command, leveraging DFlash:
- 
-```shell
-VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve --model $MODEL_CKPT \
-    --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
-    --moe-backend humming \
-    --max-num-seqs 512 \
-    --trust-remote-code \
-    --max-model-len 1048576 \
-    --max-num-batched-tokens 32768 \
-    --enable-prefix-caching \
-    --async-scheduling \
-    --quantization modelopt_fp4 \
-    --speculative_config.method dflash \
-    --speculative_config.model $DFLASH_CKPT \
-    --speculative_config.num_speculative_tokens 3 \
-    --mamba-backend flashinfer \
-    --mamba-ssm-cache-dtype float16 \
-    --enable-mamba-cache-stochastic-rounding \
-    --mamba-cache-philox-rounds 5 \
-    --mamba-cache-mode align
+    --mamba-ssu-algorithm simple \
+    --reasoning-parser nemotron_v3 \
+    --tool-call-parser qwen3_coder \
+    --enable-auto-tool-choice
 ```
  
-* **Context Length:** The H100 and GB200 snippets above serve the model's full 1M-token context window by default. If you're memory-constrained — or want more KV-cache headroom at high concurrency — lower `--max-model-len` to match your workload and drop `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`. 
+* **Context Length:** The H100 and GB200 snippets above serve the model's full 1M-token context window by default. If you're memory-constrained — or want more KV-cache headroom at high concurrency — lower `--max-model-len` to match your workload.
 
 ### **TensorRT-LLM**
- 
-Tested Version (July 18th Nightly):
-* Nightly commit: `v1.3.0rc21`
+
+> For more indepth instructions on how to deploy through TensorRT-LLM, head [here](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/usage-cookbook/Nemotron-3.5-Lightning/trtllm_cookbook.ipynb)
+
+Container: `nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24`
+
 #### **1x H100**
- 
-**1M context - 1x H100, BS=8, NVFP4 (MARLIN):**
- 
+
 ```shell
-cat > ./extra-llm-api-config.yml << EOF
+cat > nemotron-35-lightning-nvfp4-mtp.yaml << EOF
 kv_cache_config:
   dtype: fp8
   enable_block_reuse: false
-  mamba_state_cache_interval: 8192
+  mamba_state_config:
+     periodic_snapshot_interval: 8192
   free_gpu_memory_fraction: 0.8
   mamba_ssm_cache_dtype: float16
   mamba_ssm_stochastic_rounding: true
   mamba_ssm_philox_rounds: 5
 moe_config:
    backend: MARLIN
+nvfp4_gemm_config:
+  allowed_backends: [marlin, cutlass, cublaslt, cuda_core]
 cuda_graph_config:
     enable_padding: true
-    max_batch_size: 128
+    max_batch_size: 8
+speculative_config:
+  decoding_type: MTP
+  max_draft_len: 3
+  allow_advanced_sampling: true
 enable_chunked_prefill: true
 num_postprocess_workers: 4
 print_iter_log: true
 stream_interval: 10
 disable_overlap_scheduler: false
 EOF
- 
-trtllm-serve  \
-<your_nvfp4_ckpt_path> \
---max_batch_size 128 \
+
+trtllm-serve \
+$MODEL_CKPT \
+--max_batch_size 8 \
 --max_num_tokens 8192 \
---trust_remote_code \
 --reasoning_parser nemotron-v3 \
 --tool_parser qwen3_coder \
---extra_llm_api_options extra-llm-api-config.yml
+--config nemotron-35-lightning-nvfp4-mtp.yaml
 ```
- 
-* **Long-context configuration:** For long-context benchmarking, set `TLLM_ALLOW_LONG_MAX_MODEL_LEN=1` as an environment variable and add `--max_seq_len <seq_len>` as the desired maximum context length (up to 1048576).
+
+* **Context length:** The command above serves the model's full 1M-token context window by default. If you're memory-constrained — or want more KV-cache headroom at higher concurrency — lower `--max_seq_len` to match your workload.
+
 ### **SGLang**
- 
-Tested Version (July 18th Nightly):
-* Nightly commit: `99f5a6f`
+
+> For more indepth instructions on how to deploy through SGLang, head [here](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/usage-cookbook/Nemotron-3.5-Lightning/sglang_cookbook.ipynb)
+
+* Container: `lmsysorg/sglang:dev-nemotron3-5-lighting`
+
 #### **1x H100**
- 
-**1M context - 1x H100, BS=8, NVFP4:**
- 
+
 ```shell
-export PORT=8000
-sglang serve --model-path $MODEL_CKPT \
-    --port $PORT \
+sglang serve \
+    --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
+    --max-running-requests 256 \
     --trust-remote-code \
-    --enable-metrics \
-    --tensor-parallel-size 1 \
-    --pipeline-parallel-size 1 \
-    --data-parallel-size 1
-```
- 
-* **Context length:** To use up to 1M, set `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1` and `--context-length 1048576`.
-### **Llama.cpp**
- 
-Run Lightning 3.5 locally on consumer GPUs. The walkthrough below builds llama.cpp with DFlash support, converts the target and draft models to GGUF (with optional quantization), and serves an OpenAI-compatible endpoint on a single RTX 5090 — including a speed-bench harness to verify throughput.
- 
-<details>
-<summary><b>1x RTX 5090 — DFlash: Click to expand!</b></summary>
-It is assumed you possess the BF16 target model directory and BF16 draft model directory.
- 
-> The server examples below use `$PORT` — set `$PORT = 8000` to align with the [API Client](#api-client) examples.
- 
-First, clone the llama.cpp repo:
- 
-```shell
-git clone {{<!-- INSERT: public llama.cpp repo URL and branch (checklist 8) -->}}
-```
- 
-Build the repo:
- 
-```powershell
-cd $LLAMA_CPP
- 
-cmake -S . -B build-cuda -G Ninja `
-  -DCMAKE_BUILD_TYPE=Release `
-  -DGGML_CUDA=ON
- 
-cmake --build build-cuda --config Release -j 18
-```
- 
-Convert the models to `.gguf` format if not done already:
- 
-```powershell
-# for target model
-$PYTHON convert_hf_to_gguf.py $TARGET_HF `
-  --outtype bf16 `
-  --outfile "$OUT\nemotron-3.5-lightning-target-bf16.gguf"
- 
-# for draft model. Note that it requires the corresponding target model as an
-# additional argument.
-$PYTHON convert_hf_to_gguf.py $DFLASH_HF `
-  --target-model-dir $TARGET_HF `
-  --outtype bf16 `
-  --outfile "$OUT\nemotron-3.5-dflash-bf16.gguf"
- 
-# one can also take NVFP4 models from HF and directly convert them into GGUF
-# using the following command
-$PYTHON convert_hf_to_gguf.py $MODEL_PATH `
-  --fp8-as-q8 `
-  --outfile model.gguf
-```
- 
-If required, you can also quantize the BF16 model:
- 
-```powershell
-"$LLAMA_CPP\build-cuda\bin\llama-quantize.exe" `
-  "to\bf16\model.gguf" `
-  "to\quantized\model.gguf" `
-  Q4_K_M
- 
-"$LLAMA_CPP\build-cuda\bin\llama-quantize.exe" `
-  "$OUT\nemotron-3.5-dflash-bf16.gguf" `
-  "$OUT\nemotron-3.5-dflash-Q4_K_M.gguf" `
-  Q4_K_M
-```
- 
-For a target-only run:
- 
-```powershell
-"$LLAMA_CPP\build-cuda\bin\llama-server.exe" `
-  -m "to\target\model.gguf" `
-  --temp 0 -s 421 --top-k 1 `
-  -np 1 `
-  -c 40960 `
-  --port $PORT `
-  -ngl 99 `
-  -fa on `
-  --jinja `
-  --no-webui `
-  --fit off
-```
- 
-For a run with DFlash:
- 
-```powershell
-"$LLAMA_CPP\build-cuda\bin\llama-server.exe" `
-  -m "to\target\model.gguf" `
-  -md "to\draft\model.gguf" `
-  --spec-type draft-dflash `
-  --spec-draft-n-max 7 `
-  --temp 0 -s 421 --top-k 1 `
-  -np 1 `
-  -c 40960 `
-  --port $PORT `
-  -ngl 99 `
-  -fa on `
-  --jinja `
-  --no-webui `
-  --fit off
-```
- 
-You can vary the amount of draft tokens generated at once by changing the `--spec-draft-n-max` flag.
- 
-Once you've got a server started up (it won't take long with llama.cpp), open another terminal to query it with the following:
- 
-```powershell
-# run this once
-pip install -r "$LLAMA_CPP\tools\server\bench\speed-bench\requirements.txt"
- 
-# do a hf login, if the following command doesn't work.
-$PYTHON "$LLAMA_CPP\tools\server\bench\speed-bench\speed_bench.py" `
-  --url http://127.0.0.1:$PORT `
-  --bench qualitative `
-  --category all `
-  --osl 128 `
-  --concurrency 1 `
-  --output "$OUT\result.json"
+    --chunked-prefill-size 32768 \
+    --mem-fraction-static 0.9 \
+    --speculative-algorithm EAGLE \
+    --speculative-draft-model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
+    --speculative-num-steps 3 \
+    --speculative-eagle-topk 1 \
+    --speculative-num-draft-tokens 4 \
+    --mamba-backend flashinfer \
+    --mamba-radix-cache-strategy extra_buffer \
+    --reasoning-parser nemotron_3 \
+    --tool-call-parser qwen3_coder
 ```
- 
-</details>
+
+* **Context length:** The command above serves the model's full 1M-token context window by default. If you're memory-constrained — or want more KV-cache headroom at higher concurrency — set `--context-length` to a smaller value.
 
 ### **API Client**
 
-The examples below use the OpenAI-compatible client and work with any of the serving backends above. All backends serve on port `8000` (vLLM and TRT-LLM by default; SGLang and llama.cpp via `$PORT=8000`), so the `base_url` works as-is. Recommended sampling settings are **Temperature 1.0** and **Top_P 0.95**.
+The examples below use the OpenAI-compatible client and work with any of the serving backends above. All backends serve on port `8000` (vLLM and TRT-LLM by default; SGLang via `$PORT=8000`), so the `base_url` works as-is. Recommended sampling settings are **Temperature 1.0** and **Top_P 0.95**.
 
 The vLLM snippets above register the model as `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` via `--served-model-name`. For the other backends — or if you change that flag — copy the identifier returned by `GET /v1/models` into `MODEL` below.
 
@@ -672,18 +542,6 @@ response = client.chat.completions.create(
 print(response.choices[0].message.tool_calls)
 ```
 
-## Inference
-
-* **Acceleration Engine:** PyTorch
-* **Test Hardware:**
-    * NVIDIA Hopper
-      - 1-8x H100
-      - 1-8x H200
-    * NVIDIA Blackwell
-      - GB200
-      - DGX Spark (GB10)
-      - GeForce RTX 5090
-
 ## Training, Testing, and Evaluation Datasets
 
 # Training
@@ -694,8 +552,8 @@ print(response.choices[0].message.tool_calls)
 **Time period for training data collection:** 2013 to December 2025
 **Time period for testing data collection:** 2013 to December 2025
 **Time period for validation data collection:** 2013 to December 2025
-**Data Collection Method by dataset:** Hybrid: Automated, Human, Synthetic
-**Labeling Method by dataset:** Hybrid: Automated, Human, Synthetic
+**Data Collection Method by dataset:** Hybrid: Automated, Manually-Collected, Synthetic
+**Labeling Method by dataset:** Hybrid: Automated, Manually-Labeled, Synthetic
 
 NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 is pre-trained on a large corpus of high-quality curated and synthetically-generated data. It is trained in the English language, as well as 19 other spoken languages and 43 programming languages. Our sources cover a variety of document types such as: webpages, dialogue, articles, and other written materials. The corpus spans domains including legal, math, science, finance, and more. We also include a small portion of question-answering, and alignment style data to improve model accuracy. The model was pre-trained for more than 20 trillion tokens.
 
@@ -819,6 +677,7 @@ The GitHub Crawl was collected using the GitHub REST API and the Amazon S3 API.
 | Scale HLE | Unknown |
 | HackerRank Coding | Unknown |
 | RL data for Search | Gemini 3; GPT-5 |
+| Mercor SWE-AgentsV1 | Undisclosed |
 
 ## Private Non-publicly Accessible Datasets by NVIDIA
 
@@ -1019,23 +878,37 @@ For our post-training recipe, we focused on the following languages in addition
 ## Testing Datasets:
 
 **Data Collection Method by dataset** <br>
-* Hybrid: Automated, Human, Synthetic
+* Hybrid: Automated, Manually-Collected, Synthetic
 **Labeling Method by dataset** <br>
-* Hybrid: Automated, Human, Synthetic
+* Hybrid: Automated, Manually-Labeled, Synthetic
 **Properties:** This corpus comprises a mix of high-quality standard benchmarks and test suites for modern agentic AI. These benchmarks test model capabilities on tasks such as tool-calling and instruction following.
 
 ## Evaluation Datasets:
 
 **Data Collection Method by dataset** <br>
-* Hybrid: Automated, Human, Synthetic
+* Hybrid: Automated, Manually-Collected, Synthetic
 **Labeling Method by dataset** <br>
-* Hybrid: Automated, Human, Synthetic
+* Hybrid: Automated, Manually-Labeled, Synthetic
 **Properties:** This corpus comprises a mix of high-quality standard benchmarks and test suites for modern agentic AI. These benchmarks test model capabilities on tasks such as tool-calling and instruction following.
 
+## Inference
+
+* **Acceleration Engine:** PyTorch
+* **Test Hardware:**
+    * NVIDIA Hopper
+      - 1-8x H100
+      - 1-8x H200
+    * NVIDIA Blackwell
+      - GB200
+      - DGX Spark (GB10)
+      - GeForce RTX 5090
+
 ## Ethical Considerations
 
-NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+
+We advise against circumvention of any provided safety guardrails contained in the Model without a substantially similar guardrail appropriate for your use case. For more details: [Safety](./safety.md) and [Explainability](./explainability.md) Subcards.
 
-Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+For more detailed information on ethical considerations for this model, please see the Model Card++ [Bias](./bias.md), and [Privacy](./privacy.md) Subcards.
 
-Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/)
+Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

bias.md

@@ -0,0 +1,10 @@
+| Field | Response |
+| :---- | :---- |
+| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None |
+| Bias Metric (If Measured): | [BBQ Accuracy Scores in Ambiguous Contexts](https://github.com/nyu-mll/BBQ/) |
+| Which characteristic (feature) show(s) the greatest difference in performance?: | The model shows high variance in the characteristics when it is used with a high temperature. |
+| Measures taken to mitigate against unwanted bias: | Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) employed to calibrate the model’s reasoning capabilities to maintain logical consistency and appropriate complexity when interacting with or interpreting data from diverse age demographics. |
+| If using internal data, description of methods implemented in data acquisition or processing, if any, to address the prevalence of identifiable biases in the training, testing, and validation data: | The training datasets contain a large amount of synthetic data generated by LLMs. We manually curated prompts. |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | [BBQ](https://github.com/nyu-mll/BBQ/) |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | These datasets, such as web-scraped finance reasoning data derived from SEC EDGAR filings, science and math problem datasets, OpenResearcher/source-document datasets, Common Crawl, CC-News, Wikimedia, and long-context document datasets, do not collectively or exhaustively represent all demographic groups (and proportionally therein). For instance, these datasets do not contain explicit mentions of demographic classes such as age, gender, or ethnicity in approximately 97% to 99.9% of finance reasoning samples and in over 85% of samples across the broader assessed datasets. In the subset where such terms are present, these datasets contain notable representational skews. For example, ethnicity mentions are often dominated by Middle Eastern contexts (found in finance documents) or "White," "Two or more," and "Black or African American" as the most frequent ethnic identifiers, while references categorized as male-only significantly outnumber those categorized as female-only. Furthermore, gender is explicitly mentioned in approximately 12% of samples across the broader dataset assessment, yet in only 0.9% of finance-specific samples. Dataset-level results vary by source type, with long-context/source-document datasets containing higher explicit demographic mention rates compared to certain web-scraped sources. To mitigate these imbalances, we recommend considering evaluation techniques such as bias audits, fine-tuning with demographically balanced datasets, and mitigation strategies such as counterfactual data augmentation to align with the desired model behavior. This evaluation used a 3,000-sample subset per dataset, identified as the optimal threshold for maximizing embedder accuracy. |
+| Unwanted Bias Testing: | Constrained to English-language inputs. Multi-lingual parity is not currently claimed or guaranteed. |

explainability.md

@@ -0,0 +1,14 @@
+| Field | Response |
+| :---- | :---- |
+| Intended Task/Domain: | Text generation, reasoning, and chat |
+| Model Type: | Text-to-text Mamba2-Transformer Hybrid |
+| Intended Users: | Generative AI creators working with conversational AI models and image content. |
+| Output: | Text |
+| Tools used to evaluate datasets to identify synthetic data and ensure data authenticity. | We used a Gemma-3 4B-based filtering model fine-tuned on [Nemotron Content Safety Dataset v2](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) to ensure the quality of synthetic data. |
+| Describe how the model works: | Generates text by predicting the next word or token based on the context provided in the input sequence using multiple self-attention layers. |
+| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Age, Disability Status, Gender Identity, Nationality, Physical Appearance, Ethnicity, Socioeconomic Status, Sexual Orientation, Religion |
+| Technical Limitations & Mitigation: | This model performs particularly well in instruction following regimes, as such may be strongly influenced by untrusted inputs and should be paired with appropriate guardrails and data filtering to better align use-case behaviors when exposed to such data. |
+| Verified to have met prescribed NVIDIA quality standards: | Yes |
+| Performance Metrics: | Accuracy, Throughput, and User-side throughput |
+| Potential Known Risks: | The model was optimized explicitly for instruction following and as such is more susceptible to prompt injection and jailbreaking in various forms as a result of its instruction tuning. This means that the model should be paired with additional rails or system filtering to limit exposure to instructions from malicious sources -- either directly or indirectly by retrieval (e.g. via visiting a website) -- as they may yield outputs that can lead to harmful, system-level outcomes up to and including remote code execution in agentic systems when effective security controls including guardrails are not in place. The model may generate answers that may be inaccurate, omit key information, include irrelevant or redundant text, or produce socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. The model may exhibit self-anthropomorphism (e.g., displaying human-like characteristics in dialogue, such as expressing preferences and emotions). |
+| Licensing: | Use of this model is governed by the [OpenMDW License Agreement, version 1.1](https://raw.githubusercontent.com/OpenMDW/OpenMDW/refs/heads/main/1.1/LICENSE.OpenMDW-1.1) (OpenMDW-1.1). |

privacy.md

@@ -0,0 +1,5 @@
+| Privacy Information |
+| :--- |
+| Nemotron 3.5 Lightning was trained on large-scale publicly available data that may contain images, audio-video, and text relating to people. NVIDIA collected and used this data in compliance with applicable data protection and privacy laws. This model was not designed to derive insights or otherwise learn from any personal data contained in the datasets. |
+| NVIDIA uses a combination of filters, data minimization techniques, and other guardrails to help prevent personal data from being recited by our models. We employ automated tools and data processing techniques during pre-training or training to identify and filter certain categories of personal data. |
+| Please review NVIDIA's [Privacy Policy](https://www.nvidia.com/en-us/about-nvidia/privacy-policy/) for more information. |

safety.md

@@ -0,0 +1,9 @@
+| Field | Response |
+| :---- | :---- |
+| Model Application Field(s): | Chat, Instruction Following, Chatbot Development, Code Generation, Reasoning, Customer Service |
+| Describe the life critical impact (if present). | Not Applicable |
+| Description of methods implemented in data acquisition or processing, if any, to address other types of potentially harmful data in the training, testing, and validation data: | We used a guard model for content safety to exclude potentially harmful data from training.  |
+| Description of any methods implemented in data acquisition or processing, if any, to address illegal or harmful content in the training data, including, but not limited to, child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII) | We used a Gemma-3 4B-based guard model trained on [Nemotron Content Safety Dataset v2](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) for content safety to exclude potentially illegal or harmful content from the training. |
+| Use Case Restrictions: | Use of this model is governed by the [OpenMDW License Agreement, version 1.1](https://raw.githubusercontent.com/OpenMDW/OpenMDW/refs/heads/main/1.1/LICENSE.OpenMDW-1.1) (OpenMDW-1.1).|
+| Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development.  Restrictions enforce dataset access during training, and dataset license constraints adhered to. |
+| This AI model was developed based on our policies to ensure responsible data handling and risk mitigation. The datasets used for training have been scanned for harmful content and illegal content, consistent with our policies including scanning for Child Sexual Abuse Material (CSAM). Ongoing review and monitoring mechanisms are in place based on our policies and to maintain data integrity. | True. We use [Nemotron Content Safety Dataset V2](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) and an internal safety dataset specialized for minority sexuality for content safety evaluation to ensure the safety of this model. |