LocateAnything-3B-AWQ-W4A16

AWQ 4-bit quantization of nvidia/LocateAnything-3B — a visual-grounding VLM (Qwen2.5-3B-Instruct LLM + Eagle/MoonViT vision encoder) for referring, detection, pointing and layout/OCR localization.

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scales (and weight clipping); activations stay BF16 Disk size: ~3.4 GB weights (vs ~7.2 GB BF16, ~2.1×) Quantized by: sahilchachra Tooling: llm-compressor AWQModifier → compressed-tensors (pack-quantized)

Note on what is quantized: only the language tower's linear weights are quantized — the 36 Qwen2 decoder layers (self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj), i.e. 252 modules. The vision encoder (vision_model.*), the vision→LLM connector (mlp1.*), the token embeddings, lm_head and all norms stay at BF16. The headline name reflects the LM-tower quantization; the on-disk size averages the int4 and BF16 parts.

Calibration (task-specific)

AWQ is calibration-based. This checkpoint was calibrated on 256 samples from lmms-lab/RefCOCOg (referring-expression grounding on COCO images) — the model's own task. Each sample is a real image paired with a ground_single-style referring prompt, so the activation statistics that drive AWQ's per-channel scaling reflect genuine grounding workloads (the LM tower sees real projected image tokens, not text-only input). Group size 128, AWQ scale search + weight clipping.

Verification (smoke test)

Loaded with transformers + trust_remote_code on an NVIDIA Thor and run on referring/detection images. The AWQ model runs and produces grounding output near-identical to the BF16 source — box coordinates differ by ~1–2 units on the 0..1000 grid (sub-pixel). Structure: 252 LM-tower modules carry weight_packed (int4) + weight_scale; vision tower, connector, embeddings and lm_head remain BF16. This is a functional smoke test, not a quality benchmark.

Usage

Load with trust_remote_code; use sdpa attention and run_compressed=False (decompress int4 → dense at load, since the custom-code path has no int4 compute kernel):

import torch
from transformers import AutoModel, AutoTokenizer, AutoProcessor, AutoConfig

repo = "sahilchachra/LocateAnything-3B-AWQ-W4A16"
cfg = AutoConfig.from_pretrained(repo, trust_remote_code=True)
for c in [cfg, getattr(cfg, "text_config", None), getattr(cfg, "vision_config", None)]:
    if c is not None:
        c._attn_implementation = "sdpa"
cfg.quantization_config["run_compressed"] = False

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, config=cfg, torch_dtype=torch.bfloat16,
                                  trust_remote_code=True).to("cuda").eval()

Then follow the base model's prompt format and .generate(...) usage (the processor needs lmdb; decord is only needed for video). Output boxes are normalized to 0..1000. See nvidia/LocateAnything-3B for full examples.

Notes

AWQ W4A16: LM-tower weights are 4-bit (symmetric, group 128); activations and all other modules stay BF16.
Format: compressed-tensors pack-quantized. All upstream custom code, processor, tokenizer and LICENSE are carried over.
Symmetric int4 was chosen for broad loadability (vLLM / transformers / compressed-tensors).
Smoke-tested only; not a quality benchmark.