V-JEPA 2 Archetypal Sparse Transcoder (block-10 FFN)
To our knowledge, the first sparse transcoder trained on a self-supervised video model's internal features. It sparsely decomposes the computation of the block-10 FFN of a frozen V-JEPA 2.1 ViT-B/16 384px encoder, using an archetypal decoder (RA-SAE-style: decoder atoms are sparse convex combinations of K-means centroids of real activations) combined with the layer-to-layer prediction objective of sparse transcoders.
Headline checkpoint from "V-JEPA 2 Video Transcoders Find Motion-Based Features" (Ghassemlou & Joseph, 1st IJCAI Workshop on Safe Physical AI, 2026).
What it does
The transcoder reads the input to the block-10 FFN (norm2(x), i.e. the
post-LayerNorm activation) and predicts that FFN's output, through a sparse
latent z:
z = ReLU(W_enc · x + b_enc) # sparse codes [N, 12288]
D = topk_softmax(A_logits, K) @ data_pool + Lambda # archetypal decoder atoms
x_hat = Dᵀ · z + b_dec # predicted FFN output [N, 768]
The K-means pool (data_pool) is bundled inside the weights, so the decoder is
fully self-contained.
Held-out performance (2M tokens, clip-disjoint 10% split of SSv2)
| Metric | Value |
|---|---|
| Explained variance (EV) | 0.480 |
| L0 (mean active features) | 59.1 |
| Alive fraction | 4.49% (552 / 12,288) |
| Reconstruction cosine | 0.727 |
A random-init control at the same architecture and same K-means pool
(random_init_control.safetensors) scores EV ≈ −1.4×10⁴, L0 ≈ 6,121
Files
| File | Description |
|---|---|
model.safetensors |
Headline transcoder weights (K-means pool bundled) |
random_init_control.safetensors |
Random-init baseline, same shape/pool |
config.json |
Architecture + preprocessing spec |
modeling_archetypal_transcoder.py |
Self-contained model class |
load_example.py |
Minimal loader + forward pass |
Usage
# needs: torch, safetensors, and modeling_archetypal_transcoder.py from this repo
from load_example import load_transcoder
m = load_transcoder("model.safetensors") # data_pool is inside the weights
x_hat, z = m(x) # x: V-JEPA-2.1 block-10 FFN *input* activations, [N, 768] fp32
Substrate + preprocessing are part of the model
The weights are only meaningful against the exact activations they were trained on. You must:
- Run the frozen V-JEPA 2.1 ViT-B/16 384px encoder
(Meta's
vjepa2_1_vit_base_384; downloaded from Meta's public CDN, its use is governed by Meta's license). - Extract the input to
blocks[10].mlp(norm2(x)) as the transcoder input; the target it predicts is that FFN's output. - Match preprocessing exactly: 384px, 16 frames, tubelet 2, patch 16, fp32,
ImageNet normalization (
mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]).
Feeding differently-normalized or wrong-layer activations produces meaningless output. The full extraction / training / evaluation / feature-viewer pipeline is in the code repo below.
Interpretability
Features were analyzed with a non-VLM, label-distribution protocol over the 174 Something-Something v2 motion templates, surfacing motion-aligned features that track a common motion primitive across different object identities. See the paper and code repo for the feature viewer and concentration analysis.
Code
https://github.com/sghassemlou/V-JEPA-2-Transcoder --> training, evaluation, and the interactive feature-viewer (HTML) code.
Citation
@inproceedings{ghassemlou2026vjepa2transcoder,
title = {V-JEPA 2 Video Transcoders Find Motion-Based Features},
author = {Ghassemlou, Susan Soraya Afshar and Joseph, Sonia},
booktitle = {1st IJCAI Workshop on Safe Physical AI},
year = {2026}
}
License
Transcoder weights released under CC-BY-4.0. Use of the underlying V-JEPA 2 model is governed by Meta's license terms.
- Downloads last month
- 37