jcbtc/Laguna-S-2.1-NVFP4-GGUF

Laguna S 2.1 NVFP4 GGUF — 256K-capable on one RTX 5090

117.6B total parameters, about 8.5B active per token, and up to a 262,144-token context on a 32 GB RTX 5090 with 64 GB system RAM.

This is Poolside's native NVFP4 Laguna S 2.1 checkpoint packaged as one 67.03 GiB GGUF for llama.cpp. Automatic tensor-aware CPU/GPU placement keeps the dense execution path and selected tensors on the GPU while offloading expert weights to RAM. It does not use DFlash or speculative decoding.

The included runtime correction connects Laguna's native NVFP4 scale tensors to every relevant graph path. The result is a practical long-context deployment for Blackwell rather than a smaller, separately requantized checkpoint.

What has been validated

Configured context	Validation	Uncached prefill	Generation	Peak VRAM	Host RAM available	Status
65,536	65,269 actual tokens; exact needle retrieval	439.82 tok/s	25.69 tok/s at full depth; 27.81 tok/s short-depth	31,092 MiB	about 9.9 GiB	Fast lane
131,072	130,800 actual tokens; exact needle retrieval	417.16 tok/s	22.51 tok/s at full depth; 23.34 tok/s short-depth	31,056 MiB	8.2 GiB occupied	Recommended
262,144	Exact 262,144-token slot allocation and coherent generation	—	18.68 tok/s short-depth	31,016 MiB	5.2 GiB idle	Maximum-context lane

The 64K and 128K rows are fully occupied, uncached semantic tests—not empty KV-cache allocations. The 256K profile admits the complete 262,144-token slot and generates coherently; a completely occupied 256K semantic pass has not yet been claimed here.

Read the speed numbers correctly

1,201.93 tok/s is the fastest measured prompt-processing/prefill result, obtained on the full 65,269-token prompt with the separately tuned batch 1024 / ubatch 1024 64K speed profile. It is not output-token speed.

Measured output generation is 18.68–27.81 tok/s, depending on configured context and actual occupied depth. Real Hermes tool-agent generation measured 27.63 tok/s.

The larger ubatch is a 64K prefill optimization, not the defining feature of this release. The balanced batch 512 / ubatch 256 profile was used for the 64K → 128K → 256K context ladder and is the recommended starting point for long-context deployment.

Quality results

Evaluation	Published result	Sampling
HermesAgent-20, corrected verifier semantics	90/100	temperature 0, thinking off
Tool-Eval, corrected score	86/100	temperature 1, thinking off
BigCodeBench Hard/Instruct	28.38%	temperature 1, top_p=1, top_k=20, min_p=0, seed 42, thinking off

The corrected Hermes score includes the advertised ripgrep dependency, adequate turn headroom, and hardened parsing for valid tool calls. The Tool-Eval and BigCodeBench runs use the recommended temperature-1 sampler. No legacy raw harness score is reported.

Model file

File	Size	SHA-256
Laguna-S-2.1-NVFP4.gguf	71,977,030,080 bytes / 67.03 GiB	5cf866a0b1531c62a6754e811b64b8bd867b9f5cd5d6a69b2cde04077d807e87

The routed expert banks account for about 59.485 GiB across 47 sparse layers. The remaining approximately 7.48 GiB contains attention, router, dense-leading, shared-expert, output, and other tensors—mostly BF16 where the source checkpoint excludes them from NVFP4. This is why the complete native checkpoint is about 67 GiB.

Memory placement

--fit on adjusts the tensor split against actual free VRAM. As the context grows, llama.cpp reserves more GPU space for KV cache and moves more model buffers into system RAM.

Context	Model buffers on GPU	Host-backed model buffers	Q8 global KV + SWA KV	Measured total GPU used
64K	about 27.4 GiB	about 41.2 GiB	1.632 GiB + 57 MiB	31,092 MiB
128K	about 25.8 GiB	about 42.8 GiB	3.264 GiB + 57 MiB	31,056 MiB
256K	about 22.5 GiB	about 46.1 GiB	6.528 GiB + 57 MiB	31,016 MiB

The buffer figures are loader-accounting approximations rather than additive partitions of the GGUF file. CUDA workspaces and loader bookkeeping are additional. The whole-device VRAM readings include a roughly 770 MiB desktop baseline.

Hardware target

The validated target is:

• NVIDIA RTX 5090 / Blackwell GPU with 32 GB VRAM
• 64 GB system RAM
• current NVIDIA driver
• CUDA 12.8 or newer; CUDA 13 recommended
• Linux

The build and commands are not specific to NixOS or to the test machine's CPU. They contain no architecture-specific CPU compiler flags or topology masks. The example thread counts are portable starting values; tune them for your CPU and memory subsystem. Faster host memory materially helps the RAM-resident expert path.

Native NVFP4 acceleration is Blackwell-specific. Do not assume the same speed or kernel behavior on pre-Blackwell CUDA, Vulkan, Metal, or CPU-only backends.

Runtime: small fork required

At the pinned upstream build, Laguna did not pass all native NVFP4 scale tensors into its attention gate, routed experts, shared experts, dense FFN, and output projection. This release therefore requires a focused source correction.

The ready-to-build runtime is:

• ciru-ai/ciruinference
• tested tag: laguna-nvfp4-b10106

It is a pinned llama.cpp source tree with the Laguna scale-wiring correction committed normally. This repository also includes patches/0001-laguna-wire-nvfp4-scale-tensors.patch and scripts/build-linux-cuda.sh as an auditable apply-on-top path. Once equivalent wiring lands upstream, the fork will no longer be necessary.

Build on mainstream Linux

Ubuntu / Debian

sudo apt-get update
sudo apt-get install -y build-essential cmake ninja-build git curl libcurl4-openssl-dev

Fedora

sudo dnf install -y gcc-c++ cmake ninja-build git curl libcurl-devel

Install the NVIDIA driver and CUDA toolkit from NVIDIA's repository, then:

nvidia-smi
nvcc --version

git clone https://github.com/ciru-ai/ciruinference.git
cd ciruinference
git checkout laguna-nvfp4-b10106

cmake -S . -B build-cuda \
  -DGGML_CUDA=ON \
  -DLLAMA_CURL=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build-cuda --config Release -j "$(nproc)" \
  --target llama-server llama-cli

The build auto-detects the local CUDA target. For a Blackwell cross-build, add -DCMAKE_CUDA_ARCHITECTURES=120 to the CMake configuration.

Download

Install the Hugging Face CLI, authenticate if required, and download the single GGUF:

python3 -m pip install -U "huggingface_hub[cli]"

hf download jcbtc/Laguna-S-2.1-NVFP4-GGUF \
  Laguna-S-2.1-NVFP4.gguf \
  --local-dir .

Recommended 128K deployment

This is the best validated balance of real occupied context and generation speed:

./build-cuda/bin/llama-server \
  --model ./Laguna-S-2.1-NVFP4.gguf \
  --ctx-size 131072 \
  --parallel 1 \
  --batch-size 512 \
  --ubatch-size 256 \
  --threads 8 \
  --threads-batch 16 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --fit on \
  --fit-target 1024 \
  --fit-ctx 131072 \
  --load-mode none \
  --jinja \
  --spec-type none \
  --no-context-shift \
  --cache-ram 0 \
  --no-cache-idle-slots \
  --metrics \
  --slots \
  --no-webui \
  --host 0.0.0.0 \
  --port 8080 \
  --alias laguna-s21-nvfp4-128k

The equivalent portable launcher is included as scripts/serve-128k.sh.

Maximum 256K deployment

The same deployment scales to the model's full 262,144-token context:

./build-cuda/bin/llama-server \
  --model ./Laguna-S-2.1-NVFP4.gguf \
  --ctx-size 262144 \
  --parallel 1 \
  --batch-size 512 \
  --ubatch-size 256 \
  --threads 8 \
  --threads-batch 16 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --fit on \
  --fit-target 1024 \
  --fit-ctx 262144 \
  --load-mode none \
  --jinja \
  --spec-type none \
  --no-context-shift \
  --cache-ram 0 \
  --no-cache-idle-slots \
  --metrics \
  --slots \
  --no-webui \
  --host 0.0.0.0 \
  --port 8080 \
  --alias laguna-s21-nvfp4-256k

Close other GPU-heavy applications before launching. On the validated machine this profile used 31,016 MiB total VRAM and left 5.2 GiB of system RAM available at idle.

The equivalent portable launcher is included as scripts/serve-256k.sh.

The OpenAI-compatible endpoint is:

http://127.0.0.1:8080/v1

Example request using the recommended tool-evaluation sampler:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "laguna-s21-nvfp4-128k",
    "messages": [{"role": "user", "content": "Write a fast parallel file indexer in Rust."}],
    "temperature": 1,
    "top_p": 1,
    "max_tokens": 1024,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Thinking is supported by the model and was explicitly disabled for the published agent, tool, and code evaluations.

Optional 64K prefill-speed profile

For workloads dominated by large prompt ingestion rather than maximum context, change the 128K command to:

--ctx-size 65536
--fit-ctx 65536
--batch-size 1024
--ubatch-size 1024
--alias laguna-s21-nvfp4-64k

That exact profile measured 1,201.93 prefill tok/s on 65,269 uncached prompt tokens and 23.75 decode tok/s immediately afterward. It is an optional throughput lane, not the default identity of the model.

Reproducibility

• llama.cpp build 10106
• upstream base commit 1425386fd996511e1f3295e7366c38289a92a271
• included Laguna NVFP4 scale-wiring correction
• RTX 5090, NVIDIA driver 610.43.02
• Q8 K/V cache
• ordinary Flash Attention on
• one sequence
• context shift off
• DFlash off
• speculative decoding off

Credits

• Base and native NVFP4 checkpoint: poolside/Laguna-S-2.1-NVFP4
• Model architecture and training: Poolside
• Runtime: ggml-org/llama.cpp
• Validated patched runtime: ciru-ai/ciruinference
• GGUF packaging, scale-wiring validation, long-context profiling, and evaluations: jcbtc

Laguna S 2.1 is released under OpenMDW-1.1. Review the upstream model card and license before deployment.