Visual Question Answering
Transformers
Safetensors
Chinese
English
qwen3_vl
image-text-to-text
multimodal
vision-language
mechanical-drawing
vqa
mechvqa
reinforcement-learning
reasoning
Instructions to use XiaofengAlg/MechVL-4B-RL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use XiaofengAlg/MechVL-4B-RL with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("visual-question-answering", model="XiaofengAlg/MechVL-4B-RL")# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("XiaofengAlg/MechVL-4B-RL") model = AutoModelForMultimodalLM.from_pretrained("XiaofengAlg/MechVL-4B-RL", device_map="auto") - Notebooks
- Google Colab
- Kaggle
XiaofengShi commited on
Upload folder using huggingface_hub
Browse files- .mdl +0 -0
- .msc +0 -0
- .mv +1 -0
- README.md +27 -25
- model-00001-of-00003.safetensors +3 -0
- model-00002-of-00003.safetensors +3 -0
- model-00003-of-00003.safetensors +3 -0
.mdl
ADDED
|
Binary file (47 Bytes). View file
|
|
|
.msc
ADDED
|
Binary file (1.23 kB). View file
|
|
|
.mv
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
Revision:master,CreatedAt:1782493251
|
README.md
CHANGED
|
@@ -1,22 +1,25 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
-
- zh
|
| 5 |
-
- en
|
| 6 |
tags:
|
| 7 |
-
- multimodal
|
| 8 |
-
- vision-language
|
| 9 |
-
- mechanical-drawing
|
| 10 |
-
- vqa
|
| 11 |
-
- mechvqa
|
| 12 |
-
- reinforcement-learning
|
| 13 |
-
- reasoning
|
| 14 |
-
base_model:
|
| 15 |
-
- MonteXiaofeng/MechVL-4B-SFT
|
| 16 |
pipeline_tag: visual-question-answering
|
| 17 |
library_name: transformers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
---
|
| 19 |
-
|
| 20 |
# MechVL-4B-RL
|
| 21 |
|
| 22 |
The **RL checkpoint** of **MechVL** — the domain-specialized multimodal model for mechanical engineering drawing understanding, introduced in:
|
|
@@ -24,11 +27,11 @@ The **RL checkpoint** of **MechVL** — the domain-specialized multimodal model
|
|
| 24 |
> **MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding** (ICML 2026)
|
| 25 |
|
| 26 |
[](https://arxiv.org/abs/2605.30794)
|
| 27 |
-
[**: DAPO over the full MechVQA training split.
|
| 34 |
- **Stage 2 (targeted)**: DAPO over a re-sampled subset with an increased proportion of underperforming subtasks.
|
|
@@ -41,7 +44,7 @@ Reward = **Accuracy** (LLM-as-a-Judge, semantic equivalence in [0,1]) + **Format
|
|
| 41 |
| Architecture | Qwen3VLForConditionalGeneration |
|
| 42 |
| Stage | 2 / 2 — RL (DAPO self-play, on top of SFT) |
|
| 43 |
| MechVQA Total | **84.85** (best across open- & closed-source MLLMs) |
|
| 44 |
-
| SFT checkpoint | [
|
| 45 |
|
| 46 |
## Output format
|
| 47 |
|
|
@@ -53,21 +56,20 @@ The model reasons then answers, enclosed as:
|
|
| 53 |
|
| 54 |
Parse the `<answer>...</answer>` span for the final answer (a regex like `<answer>(.*?)</answer>` works).
|
| 55 |
|
| 56 |
-
## Usage (
|
| 57 |
|
| 58 |
```python
|
| 59 |
import re, torch
|
| 60 |
-
from
|
| 61 |
|
| 62 |
-
model =
|
| 63 |
-
"
|
| 64 |
)
|
| 65 |
-
processor = AutoProcessor.from_pretrained("
|
| 66 |
|
| 67 |
question = "图纸中标注的零件总长度是多少?"
|
| 68 |
-
# RL format prompt (matches training): question + <think>/<answer> instruction
|
| 69 |
messages = [{"role": "user", "content": [
|
| 70 |
-
{"type": "image", "
|
| 71 |
{"type": "text", "text": question},
|
| 72 |
]}]
|
| 73 |
inputs = processor.apply_chat_template(
|
|
@@ -81,7 +83,7 @@ m = re.search(r"<answer>(.*?)</answer>", text, re.DOTALL)
|
|
| 81 |
print("answer:", m.group(1).strip() if m else text)
|
| 82 |
```
|
| 83 |
|
| 84 |
-
For
|
| 85 |
|
| 86 |
## Results
|
| 87 |
|
|
@@ -95,7 +97,7 @@ On the MechVQA benchmark (Total score):
|
|
| 95 |
| MechVL-4B-SFT | 76.36 |
|
| 96 |
| **MechVL-4B-RL (this model)** | **84.85** |
|
| 97 |
|
| 98 |
-
See [§6 of the paper](https://arxiv.org/abs/2605.30794) for the full table and ablations
|
| 99 |
|
| 100 |
## Citation
|
| 101 |
|
|
@@ -113,4 +115,4 @@ See [§6 of the paper](https://arxiv.org/abs/2605.30794) for the full table and
|
|
| 113 |
|
| 114 |
## License
|
| 115 |
|
| 116 |
-
Apache-2.0.
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
+
- zh
|
| 5 |
+
- en
|
| 6 |
tags:
|
| 7 |
+
- multimodal
|
| 8 |
+
- vision-language
|
| 9 |
+
- mechanical-drawing
|
| 10 |
+
- vqa
|
| 11 |
+
- mechvqa
|
| 12 |
+
- reinforcement-learning
|
| 13 |
+
- reasoning
|
|
|
|
|
|
|
| 14 |
pipeline_tag: visual-question-answering
|
| 15 |
library_name: transformers
|
| 16 |
+
base_model:
|
| 17 |
+
- xiaofengalg/MechVL-4B-SFT
|
| 18 |
+
frameworks: PyTorch
|
| 19 |
+
base_model_relation: finetune
|
| 20 |
+
tasks:
|
| 21 |
+
- image-text-to-text
|
| 22 |
---
|
|
|
|
| 23 |
# MechVL-4B-RL
|
| 24 |
|
| 25 |
The **RL checkpoint** of **MechVL** — the domain-specialized multimodal model for mechanical engineering drawing understanding, introduced in:
|
|
|
|
| 27 |
> **MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding** (ICML 2026)
|
| 28 |
|
| 29 |
[](https://arxiv.org/abs/2605.30794)
|
| 30 |
+
[](https://github.com/xiaofengShi/MechVQA)
|
| 31 |
|
| 32 |
## Model description
|
| 33 |
|
| 34 |
+
MechVL-4B-RL is obtained by further optimizing [MechVL-4B-SFT](https://modelscope.cn/models/xiaofengalg/MechVL-4B-SFT) with **DAPO** (Decoupled Clip and Dynamic Sampling Policy Optimization) in a **two-stage self-play** regime:
|
| 35 |
|
| 36 |
- **Stage 1 (full)**: DAPO over the full MechVQA training split.
|
| 37 |
- **Stage 2 (targeted)**: DAPO over a re-sampled subset with an increased proportion of underperforming subtasks.
|
|
|
|
| 44 |
| Architecture | Qwen3VLForConditionalGeneration |
|
| 45 |
| Stage | 2 / 2 — RL (DAPO self-play, on top of SFT) |
|
| 46 |
| MechVQA Total | **84.85** (best across open- & closed-source MLLMs) |
|
| 47 |
+
| SFT checkpoint | [xiaofengalg/MechVL-4B-SFT](https://modelscope.cn/models/xiaofengalg/MechVL-4B-SFT) |
|
| 48 |
|
| 49 |
## Output format
|
| 50 |
|
|
|
|
| 56 |
|
| 57 |
Parse the `<answer>...</answer>` span for the final answer (a regex like `<answer>(.*?)</answer>` works).
|
| 58 |
|
| 59 |
+
## Usage (ModelScope)
|
| 60 |
|
| 61 |
```python
|
| 62 |
import re, torch
|
| 63 |
+
from modelscope import AutoModelForCausalLM, AutoProcessor
|
| 64 |
|
| 65 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 66 |
+
"xiaofengalg/MechVL-4B-RL", torch_dtype=torch.bfloat16, device_map="auto"
|
| 67 |
)
|
| 68 |
+
processor = AutoProcessor.from_pretrained("xiaofengalg/MechVL-4B-RL")
|
| 69 |
|
| 70 |
question = "图纸中标注的零件总长度是多少?"
|
|
|
|
| 71 |
messages = [{"role": "user", "content": [
|
| 72 |
+
{"type": "image", "image": "path/to/drawing.png"},
|
| 73 |
{"type": "text", "text": question},
|
| 74 |
]}]
|
| 75 |
inputs = processor.apply_chat_template(
|
|
|
|
| 83 |
print("answer:", m.group(1).strip() if m else text)
|
| 84 |
```
|
| 85 |
|
| 86 |
+
Also available on [HuggingFace](https://huggingface.co/MonteXiaofeng/MechVL-4B-RL). For batch vLLM inference with the exact training-time format prompt, see [`scripts/batch_infer.py`](https://github.com/xiaofengShi/MechVQA/blob/main/scripts/batch_infer.py) (MODE=rl) and [`prompts/mech_r1.jinja`](https://github.com/xiaofengShi/MechVQA/blob/main/prompts/mech_r1.jinja).
|
| 87 |
|
| 88 |
## Results
|
| 89 |
|
|
|
|
| 97 |
| MechVL-4B-SFT | 76.36 |
|
| 98 |
| **MechVL-4B-RL (this model)** | **84.85** |
|
| 99 |
|
| 100 |
+
See [§6 of the paper](https://arxiv.org/abs/2605.30794) for the full table and ablations.
|
| 101 |
|
| 102 |
## Citation
|
| 103 |
|
|
|
|
| 115 |
|
| 116 |
## License
|
| 117 |
|
| 118 |
+
Apache-2.0.
|
model-00001-of-00003.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cf62f4c722682790123aca7c6462cc5187528f5f786777d8a5fa6503e8f39332
|
| 3 |
+
size 4408849992
|
model-00002-of-00003.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5146487eb042298747a29ff76fced3a19baade14092405ddc9ff2674046fbc0a
|
| 3 |
+
size 4466861016
|
model-00003-of-00003.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8a1dc883e795e9879df8813e10e73e3907d13ccc87413a6e5db2b664d2e68838
|
| 3 |
+
size 777920752
|