TrajGazeMerge β checkpoints
Stage-1 trajectory encoder + Stage-2 LoRA-finetuned Qwen2.5-VL-7B-Instruct checkpoints for the TrajGazeMerge project β a trajectory-aware visual token selection method that compresses egocentric visual tokens 10Γ (merge_ratio = 0.9) while keeping the kept tokens task-relevant.
Code: https://github.com/Jaewoo97/trajgaze (branch cf-mask-augmented-training).
Paper draft: docs/NeurIPS_2026_Gaze_hand_Trajectory_Merging_for_Efficient_Egocentric_Video_Understanding.pdf in the source repo.
Best result (one sentence)
TAS-only is best in both 2-dataset (StreamGaze + EgoGazeVQA) and 3-dataset (+ HD-EPIC) setups. Adding ATR / CGM losses hurts mean accuracy; counterfactual-mask diagnosis (see eval_results/) shows these extensions do not change LLM language-prior usage on EgoGazeVQA. See the code repo's docs/paper_narrative_v3.md for the mechanism-by-role reading.
| Setup | Method | StreamGaze | EgoGazeVQA | HD-EPIC | mean |
|---|---|---|---|---|---|
| A (2 ds) | TAS-only β | 67.49 | 57.77 | β | 62.63 |
| A | TAS+ATR+CGM (FULL) | 61.98 | 59.40 | β | 60.69 |
| A | CGM-only | 62.74 | 55.92 | β | 59.33 |
| B (3 ds) | TAS-only-hdepic β | 63.69 | 55.92 | 50.12 | 56.57 |
| B | TAS+ATR-hdepic | 60.65 | 54.76 | 50.66 | 55.35 |
| B | CF-1 (cf-mask aug, early-stopped) | β | β | β | ~54.5 |
| B | CF-3 (cf-mask + shuf aug, early-stopped) | β | β | β | ~50.0 |
Files
stage1/ β Trajectory encoder (Stage-1 pretrained)
| File | Size | Role |
|---|---|---|
stage1/E1_combined_AB_TAS.pth |
142 MB | A + B + TAS encoder (Stage-1 supervised). Required as --stage1-ckpt for every Stage-2 / cf-mask run. |
stage2/ β LoRA-finetuned Stage-2 (Qwen-VL + TAS encoder fine-tune)
LoRA: r=16, Ξ±=32, target_modules=[q_proj, k_proj, v_proj, o_proj]. Each .pth contains lora_state, encoder_state, and (when applicable) atr_state.
| File | Size | Stage-2 loss | Setup | Notes |
|---|---|---|---|---|
stage2/E1_combined_TAS_only.pth β
|
16 GB | CE only | A | Headline (Setup A best). |
stage2/E1_combined_TAS_ATR.pth |
16 GB | CE + ATR | A | Ablation. |
stage2/E1_combined_TAS_ATR_CGM.pth (FULL) |
16 GB | CE + ATR + CGM | A | Ablation; EgoGazeVQA mask_kept sign-flip but StreamGaze regress. |
stage2/E1_combined_CGM_only.pth |
16 GB | CE + CGM | A | Ablation; trained from earlier non-TAS Stage-1. |
stage2/E1_combined_TASonly_hdepic.pth β
|
16 GB | CE only | B | Setup B best (+ HD-EPIC). |
stage2/E1_combined_TAS_ATR_hdepic.pth |
16 GB | CE + ATR | B | Setup B ablation row. |
stage2/E1_combined_cf1_hdepic.pth |
16 GB | CE + cf-mask margin | B | Direction A (CF-1). Training stopped early at step ~6500. |
stage2/E1_combined_cf3_hdepic.pth |
16 GB | CE + cf-mask + shuf margins | B | Direction A (CF-3). Training stopped early at step ~6500. |
eval_results/ β counterfactual-mask summaries + per-sample parquets
Used to reproduce the 4Γ2 cf-mask matrix in docs/visual_grounding_diagnosis_v2.md. Each ckpt has per-variant *_mask_summary.json + *_per_sample.parquet for {streamgaze, egovqa}.
How to use
- Clone the code:
git clone https://github.com/Jaewoo97/trajgaze && cd trajgaze && git checkout cf-mask-augmented-training. - Place the Stage-1 ckpt at
TrajGaze_v2/checkpoints/E1_combined_AB_TAS/best.pth. - Place each Stage-2 ckpt at
TrajGazeMerge/checkpoints/<run-name>/best.pthmatching the file name above (drop the.pthsuffix in the dir name; the file inside is alwaysbest.pth). - Run cf-mask diagnostic (no training needed for inference):
PYTHONPATH=. python -m TrajGazeMerge.eval.counterfactual_mask_eval \ --stage1-ckpt TrajGaze_v2/checkpoints/E1_combined_AB_TAS/best.pth \ --lora-ckpt TrajGazeMerge/checkpoints/E1_combined_TAS_only/best.pth \ --val-dataset streamgaze --tag my_run
See repo README.md for full training commands (including CF-1, CF-3 reproduce commands in Β§4d).