factorjepa-surgery-noDI-head-vjepa21-vitG-2B-poc

FactorJEPA — V-JEPA 2.1 ViT-G (2B) adapted on Indian-context urban driving / walking / monument clips.

This is the surgery_noDI_head arm of the iter18 FactorJEPA ablation: surgery HEAD variant: encoder+predictor FROZEN at pretrain init; only the motion_aux head trains (noDI mixture). The study compares factor-surgery against strong fine-tuning baselines on the claim vjepa_surgery >> vjepa_pretrain >> vjepa_frozen for motion / temporal features on Indian urban video. Every non-pretrain arm initializes from the SAME continual-pretrain checkpoint (fair duel — identical data, identical starting weights).

🎯 Training summary

Field	Value
Base model	facebook/v-jepa-2-vitg
Stage	surgery_noDI_head
Architecture	V-JEPA 2.1 ViT-G (~2B params, 1664-dim, 48 layers, hierarchical concat 6656-dim)
Training data	Indian-context urban clips (10k POC pool, leakage-safe train/val/test split)
Mode	poc
Max epochs	2
Total steps	484
Batch size	32
Train / val clips	7,724 / 451
Best epoch (head val-loss)	1 (loss 2.0962)
Trainable head params	435,490
Mode mixture (L/A/I)	{"L": 0.5, "A": 0.5, "I": 0.0}
Wall time	2.9 h
Encoder	FROZEN at pretrain init (head-only training)

📈 Training trajectory (initial → final, from probe_history.jsonl)

Metric	Description	Initial	Final	Δ
probe_top1	motion-flow 16-class probe top-1	0.257	0.271	+5.2% 📈
motion_cos	intra-vs-inter motion cosine	0.1476	0.1573	+6.6% 📈
val_jepa_loss	validation JEPA loss (L1)	0.2214	0.2160	-2.5% 📉
future_l1	future-frame L1 (per clip)	0.5369	0.5369	+0.0% ➡️
block_drift_mean	mean per-block weight drift	0.73539	1.05559	+43.5% 📈

(4 checkpoints across training.)

🧪 Held-out test evaluation (N=1825 clips · 95% BCa bootstrap CI)

Metric	Value	95% CI (±)	better
action top-1	0.4964	0.0227	↑
taxonomy F1	0.7867	—	↑
motion-cos margin	0.1346	0.0046	↑
future-frame MSE	0.5330	0.0008	↓
rollout drift	0.0077	0.0002	↓
causal L1	0.5583	0.0013	↓
t-dist error	0.0065	0.0003	↓
mask-ratio slope	0.0625	0.0005	↓
order sensitivity	0.0154	0.0013	·
teacher-free drift	0.0377	0.0005	↓

Direction: ↑ higher better · ↓ lower better · · signed diagnostic. — = not computed for this arm.

⚡ Quick start — self-contained, no other code needed

This repo ships everything: the weights, the architecture (vjepa2_src/, vendored Meta V-JEPA 2 source, MIT), and a loader. Download it and run — no private package, no separate clone.

huggingface-cli download anonymousML123/factorjepa-surgery-noDI-head-vjepa21-vitG-2B-poc --local-dir factorjepa-model
cd factorjepa-model && pip install -r requirements.txt
python load_factorjepa.py --encoder student_encoder.pt    # builds 2B ViT-G, loads, forwards (no video needed)

from load_factorjepa import load_encoder, preprocess_frames, extract_features
encoder = load_encoder("student_encoder.pt", device="cuda")   # bf16 on cuda, fp32 on cpu
clip = preprocess_frames(frames_uint8)[None]                  # (T,H,W,3) uint8 -> (1, 16, 3, 384, 384)
feats = extract_features(encoder, clip)                       # (1, 4608, 1664) token features

NATIVE V-JEPA 2.1 ViT weights — NOT transformers.VJEPA2Model (AutoModel.from_pretrained fails: different keys + no 2.1 deep-supervision head). No xformers (SDPA attention). student_encoder.pt is encoder-only — for an actual next-frame prediction heatmap also load the predictor from m09c_ckpt_best.pt (key predictor): from load_factorjepa import load_predictor; predictor = load_predictor("m09c_ckpt_best.pt", device="cuda").

🏗️ Architecture

Encoder	V-JEPA 2.1 ViT-G — embed_dim=1664, depth=48, num_heads=26, RoPE, 2B params (1.84B exact)
Input	(B, 3, T=16, 384, 384) — 16 frames, 384² center-crop, ImageNet-normalized; patch 16, tubelet 2
Tokens	8 × 24 × 24 = 4608 tokens × 1664-dim (final layer); deep-supervision concat = 4608 × 6656
Predictor	2.1 predictor — predictor_embed_dim=384, depth=24, num_heads=12, dense-loss (return_all_tokens)
Attention	scaled_dot_product_attention (SDPA) — no xformers

The exact constructor kwargs are in load_factorjepa.py (verified against the eval pipeline that produced these weights). student_encoder.pt wraps the weights under the key student_state_dict — the loader unwraps it, strips module./backbone. prefixes, and asserts ≥90% of params load (fail-loud).

📦 Files in this repo

File	Size	Purpose
student_encoder.pt	~7 GB	Inference-ready ViT-G encoder weights (key student_state_dict) — load this for features
m09c_ckpt_best.pt	~8-14 GB	Best-selected ckpt incl. predictor (key predictor) — for next-frame / JEPA prediction
load_factorjepa.py	~8 KB	Self-contained loader — build model + load weights + preprocess + forward
vjepa2_src/	~100 KB	Vendored V-JEPA 2 architecture (Meta, MIT) — the encoder/predictor classes
requirements.txt	<1 KB	Pinned deps (exact versions that load these weights; no xformers)
motion_aux_head.pt	~2 MB	Motion auxiliary head (paired with student_encoder)
training_summary.json	~2 KB	Final-step metrics
probe_history.jsonl	~few KB/step	Per-checkpoint probe + drift metrics
loss_log.{jsonl,csv}	~several KB	Per-step JEPA loss trajectory
*.png / *.pdf	~few MB	Training trajectory plots (loss, drift, probe trio)

🧪 Reproducibility

This checkpoint was produced by:

CACHE_POLICY_ALL=2 ./scripts/run_train.sh surgery_noDI_head --FULL \
    2>&1 | tee logs/surgery_noDI_head_full.log

Pipeline source: iter/iter14_surgery_on_pretrain/plan_HIGH_LEVEL.md

📜 Attribution & license — the links below are provenance/credit only, NOT a setup step

✅ 100% self-contained. Everything needed to load this model is already in THIS repo (vjepa2_src/ + load_factorjepa.py + the weights). You do not need to visit, clone, pip install, or download anything from the two links below — they are license/credit only. Loading touches no other repo.

• Adapted weights (student_encoder.pt, m09*_ckpt_best.pt, motion_aux_head.pt) — Apache-2.0 (this repo). Derived from facebook/v-jepa-2-vitg (provenance only — not needed to load).
• Vendored architecture (vjepa2_src/) — Meta Platforms' V-JEPA 2, MIT, copied unmodified from github.com/facebookresearch/vjepa2 @ 204698b (credit only — the code is already in vjepa2_src/; its MIT license is at vjepa2_src/LICENSE). © Meta Platforms, Inc. and affiliates.

📝 Citation

@misc{factorjepa2026,
  title  = {FactorJEPA: Factor-disentangled SSL for Indian-context urban video},
  author = {Wanaskar, Kapil and others},
  year   = {2026},
  note   = {HF model card auto-generated by src/utils/hf_finetuned_push.py}
}

---

Model card auto-generated by src/utils/hf_finetuned_push.py at 2026-06-12T11:10:11+00:00.