XiaofengShi commited on
Commit
20dc2ee
·
verified ·
1 Parent(s): 77efcc3

Upload folder using huggingface_hub

Browse files
.mdl ADDED
Binary file (47 Bytes). View file
 
.msc ADDED
Binary file (1.23 kB). View file
 
.mv ADDED
@@ -0,0 +1 @@
 
 
1
+ Revision:master,CreatedAt:1782493251
README.md CHANGED
@@ -1,22 +1,25 @@
1
  ---
2
  license: apache-2.0
3
  language:
4
- - zh
5
- - en
6
  tags:
7
- - multimodal
8
- - vision-language
9
- - mechanical-drawing
10
- - vqa
11
- - mechvqa
12
- - reinforcement-learning
13
- - reasoning
14
- base_model:
15
- - MonteXiaofeng/MechVL-4B-SFT
16
  pipeline_tag: visual-question-answering
17
  library_name: transformers
 
 
 
 
 
 
18
  ---
19
-
20
  # MechVL-4B-RL
21
 
22
  The **RL checkpoint** of **MechVL** — the domain-specialized multimodal model for mechanical engineering drawing understanding, introduced in:
@@ -24,11 +27,11 @@ The **RL checkpoint** of **MechVL** — the domain-specialized multimodal model
24
  > **MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding** (ICML 2026)
25
 
26
  [![arXiv](https://img.shields.io/badge/arXiv-2605.30794-b31b1b.svg)](https://arxiv.org/abs/2605.30794)
27
- [![Code](https://img.shields.io/badge/Code-GitHub-181717.svg)](https://github.com/xiaofengShi/MechVQA)
28
 
29
  ## Model description
30
 
31
- MechVL-4B-RL is obtained by further optimizing [MechVL-4B-SFT](https://huggingface.co/MonteXiaofeng/MechVL-4B-SFT) with **DAPO** (Decoupled Clip and Dynamic Sampling Policy Optimization) in a **two-stage self-play** regime:
32
 
33
  - **Stage 1 (full)**: DAPO over the full MechVQA training split.
34
  - **Stage 2 (targeted)**: DAPO over a re-sampled subset with an increased proportion of underperforming subtasks.
@@ -41,7 +44,7 @@ Reward = **Accuracy** (LLM-as-a-Judge, semantic equivalence in [0,1]) + **Format
41
  | Architecture | Qwen3VLForConditionalGeneration |
42
  | Stage | 2 / 2 — RL (DAPO self-play, on top of SFT) |
43
  | MechVQA Total | **84.85** (best across open- & closed-source MLLMs) |
44
- | SFT checkpoint | [MonteXiaofeng/MechVL-4B-SFT](https://huggingface.co/MonteXiaofeng/MechVL-4B-SFT) |
45
 
46
  ## Output format
47
 
@@ -53,21 +56,20 @@ The model reasons then answers, enclosed as:
53
 
54
  Parse the `<answer>...</answer>` span for the final answer (a regex like `<answer>(.*?)</answer>` works).
55
 
56
- ## Usage (transformers)
57
 
58
  ```python
59
  import re, torch
60
- from transformers import AutoProcessor, AutoModelForImageTextToText
61
 
62
- model = AutoModelForImageTextToText.from_pretrained(
63
- "MonteXiaofeng/MechVL-4B-RL", dtype=torch.bfloat16, device_map="auto"
64
  )
65
- processor = AutoProcessor.from_pretrained("MonteXiaofeng/MechVL-4B-RL")
66
 
67
  question = "图纸中标注的零件总长度是多少?"
68
- # RL format prompt (matches training): question + <think>/<answer> instruction
69
  messages = [{"role": "user", "content": [
70
- {"type": "image", "url": "path/to/drawing.png"},
71
  {"type": "text", "text": question},
72
  ]}]
73
  inputs = processor.apply_chat_template(
@@ -81,7 +83,7 @@ m = re.search(r"<answer>(.*?)</answer>", text, re.DOTALL)
81
  print("answer:", m.group(1).strip() if m else text)
82
  ```
83
 
84
- For **batch vLLM inference** with the exact training-time format prompt, see [`scripts/batch_infer.py`](https://github.com/xiaofengShi/MechVQA/blob/main/scripts/batch_infer.py) (MODE=rl) and [`prompts/mech_r1.jinja`](https://github.com/xiaofengShi/MechVQA/blob/main/prompts/mech_r1.jinja).
85
 
86
  ## Results
87
 
@@ -95,7 +97,7 @@ On the MechVQA benchmark (Total score):
95
  | MechVL-4B-SFT | 76.36 |
96
  | **MechVL-4B-RL (this model)** | **84.85** |
97
 
98
- See [§6 of the paper](https://arxiv.org/abs/2605.30794) for the full table and ablations (DAPO > GRPO > GSPO; two-stage self-play; reward-component ablations).
99
 
100
  ## Citation
101
 
@@ -113,4 +115,4 @@ See [§6 of the paper](https://arxiv.org/abs/2605.30794) for the full table and
113
 
114
  ## License
115
 
116
- Apache-2.0.
 
1
  ---
2
  license: apache-2.0
3
  language:
4
+ - zh
5
+ - en
6
  tags:
7
+ - multimodal
8
+ - vision-language
9
+ - mechanical-drawing
10
+ - vqa
11
+ - mechvqa
12
+ - reinforcement-learning
13
+ - reasoning
 
 
14
  pipeline_tag: visual-question-answering
15
  library_name: transformers
16
+ base_model:
17
+ - xiaofengalg/MechVL-4B-SFT
18
+ frameworks: PyTorch
19
+ base_model_relation: finetune
20
+ tasks:
21
+ - image-text-to-text
22
  ---
 
23
  # MechVL-4B-RL
24
 
25
  The **RL checkpoint** of **MechVL** — the domain-specialized multimodal model for mechanical engineering drawing understanding, introduced in:
 
27
  > **MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding** (ICML 2026)
28
 
29
  [![arXiv](https://img.shields.io/badge/arXiv-2605.30794-b31b1b.svg)](https://arxiv.org/abs/2605.30794)
30
+ [![GitHub](https://img.shields.io/badge/Code-GitHub-181717.svg)](https://github.com/xiaofengShi/MechVQA)
31
 
32
  ## Model description
33
 
34
+ MechVL-4B-RL is obtained by further optimizing [MechVL-4B-SFT](https://modelscope.cn/models/xiaofengalg/MechVL-4B-SFT) with **DAPO** (Decoupled Clip and Dynamic Sampling Policy Optimization) in a **two-stage self-play** regime:
35
 
36
  - **Stage 1 (full)**: DAPO over the full MechVQA training split.
37
  - **Stage 2 (targeted)**: DAPO over a re-sampled subset with an increased proportion of underperforming subtasks.
 
44
  | Architecture | Qwen3VLForConditionalGeneration |
45
  | Stage | 2 / 2 — RL (DAPO self-play, on top of SFT) |
46
  | MechVQA Total | **84.85** (best across open- & closed-source MLLMs) |
47
+ | SFT checkpoint | [xiaofengalg/MechVL-4B-SFT](https://modelscope.cn/models/xiaofengalg/MechVL-4B-SFT) |
48
 
49
  ## Output format
50
 
 
56
 
57
  Parse the `<answer>...</answer>` span for the final answer (a regex like `<answer>(.*?)</answer>` works).
58
 
59
+ ## Usage (ModelScope)
60
 
61
  ```python
62
  import re, torch
63
+ from modelscope import AutoModelForCausalLM, AutoProcessor
64
 
65
+ model = AutoModelForCausalLM.from_pretrained(
66
+ "xiaofengalg/MechVL-4B-RL", torch_dtype=torch.bfloat16, device_map="auto"
67
  )
68
+ processor = AutoProcessor.from_pretrained("xiaofengalg/MechVL-4B-RL")
69
 
70
  question = "图纸中标注的零件总长度是多少?"
 
71
  messages = [{"role": "user", "content": [
72
+ {"type": "image", "image": "path/to/drawing.png"},
73
  {"type": "text", "text": question},
74
  ]}]
75
  inputs = processor.apply_chat_template(
 
83
  print("answer:", m.group(1).strip() if m else text)
84
  ```
85
 
86
+ Also available on [HuggingFace](https://huggingface.co/MonteXiaofeng/MechVL-4B-RL). For batch vLLM inference with the exact training-time format prompt, see [`scripts/batch_infer.py`](https://github.com/xiaofengShi/MechVQA/blob/main/scripts/batch_infer.py) (MODE=rl) and [`prompts/mech_r1.jinja`](https://github.com/xiaofengShi/MechVQA/blob/main/prompts/mech_r1.jinja).
87
 
88
  ## Results
89
 
 
97
  | MechVL-4B-SFT | 76.36 |
98
  | **MechVL-4B-RL (this model)** | **84.85** |
99
 
100
+ See [§6 of the paper](https://arxiv.org/abs/2605.30794) for the full table and ablations.
101
 
102
  ## Citation
103
 
 
115
 
116
  ## License
117
 
118
+ Apache-2.0.
model-00001-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf62f4c722682790123aca7c6462cc5187528f5f786777d8a5fa6503e8f39332
3
+ size 4408849992
model-00002-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5146487eb042298747a29ff76fced3a19baade14092405ddc9ff2674046fbc0a
3
+ size 4466861016
model-00003-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8a1dc883e795e9879df8813e10e73e3907d13ccc87413a6e5db2b664d2e68838
3
+ size 777920752