Qwen3.5-9B-Base-Nemotron-SFT-2560-steps

Model Description

This model is Qwen/Qwen3.5-9B-Base fine-tuned with full-parameter SFT on the NVIDIA Nemotron-Terminal-Corpus, following the training recipe from the paper On Data Engineering for Scaling LLM Terminal Capabilities (NVIDIA, 2026). This checkpoint corresponds to step 2560 out of 2 epochs.

Training Data

• Dataset: nvidia/Nemotron-Terminal-Corpus
• Description: ~366k terminal/coding instruction samples curated by NVIDIA to improve LLM capability on shell, CLI, and code tasks.

Training Details

Framework

• Training Framework: ms-swift
• Backend: DeepSpeed ZeRO-3

Hyperparameters

Parameter	Value
Learning Rate	2e-5
Weight Decay	1e-4
Epochs	2
Global Batch Size	128
Micro-batch Size per GPU	1
Gradient Accumulation Steps	2
Max Sequence Length	32,768 tokens
Optimizer	AdamW (β₁=0.9, β₂=0.95)
LR Scheduler	Cosine with 10% warmup
Gradient Clipping	1.0
Attention	FlashAttention-2
dtype	bfloat16

Hardware

Item	Value
GPU	H200
Nodes	8
GPUs per Node	8
Total GPUs	64

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ZhuofengLi/Qwen3.5-9B-Base-Nemotron-SFT-2560-steps",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ZhuofengLi/Qwen3.5-9B-Base-Nemotron-SFT-2560-steps")