joerowell baranowskiadam commited on
Commit
2cd4c58
·
1 Parent(s): 38e2b7e

Update README.md (#5)

Browse files

- Update README.md (5f4568e73cdf4475d5ee54e9f0d3c4a1e2d08410)


Co-authored-by: Adam Baranowski <baranowskiadam@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -88,11 +88,13 @@ Laguna M.1 has upstream support in vLLM, SGLang, and TRT-LLM thanks to the suppo
88
 
89
  #### vLLM
90
 
91
- The full vLLM recipe is on the main [Laguna M.1 model card](https://huggingface.co/poolside/Laguna-M.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-M.1-FP8` substituted for the model ID. No extra flags required.
92
 
93
  ```shell
94
  pip install 'vllm>=0.21.0'
95
 
 
 
96
  vllm serve \
97
  --model poolside/Laguna-M.1-FP8 \
98
  --tool-call-parser poolside_v1 \
 
88
 
89
  #### vLLM
90
 
91
+ The full vLLM recipe is on the main [Laguna M.1 model card](https://huggingface.co/poolside/Laguna-M.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-M.1-FP8` substituted for the model ID. Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` when serving with vLLM.
92
 
93
  ```shell
94
  pip install 'vllm>=0.21.0'
95
 
96
+ export VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0
97
+
98
  vllm serve \
99
  --model poolside/Laguna-M.1-FP8 \
100
  --tool-call-parser poolside_v1 \