G1c FINO-style adapter: metadata guidance offsets SSL-globalization directionally (+0.014) but no CI-significant recovery; F1b not de-risked at adapter scale (honest)

RESEARCH_OVERVIEW.md

@@ -59,13 +59,15 @@ geometry — no training, no metadata. It addresses none of covtoken's contribut
 localization, the rank-vs-concentration law, the conformal certificate, rare-structure focus).
 Three implications:
 - **Differentiate** in related work: frozen geometric probe + law + certificate, not adaptation.
-- **De-risks F1b**: G1a showed the eroded subspace cannot be recovered *training-free* (globalization
-  is entangling). FINO is an existence proof that *with* a training signal one can "preserve
-  informative, suppress spurious" factors — so the practical concentration-preserving lever is
-  **metadata-guided adaptation**, not steering.
-- **Complementary experiment** (new future direction): run the covtoken probe on a FINO-adapted
-  backbone — does metadata-guided adaptation preserve the mid-layer concentration subspace at depth
-  and raise rare-signal separability? Bridges the two lines and rides their release.
+- **F1b lever — TESTED at adapter scale (G1c), not de-risked.** No FINO checkpoint is released, so we
+  trained a label-free metadata-guided adapter (the trained counterpart to G1a). Result: SSL-only
+  adaptation *erodes* localizability (globalization, reproduced by a trained objective), and metadata
+  guidance *offsets* it directionally (+0.014 mean vs SSL) — but the gap is not CI-significant and it
+  does **not recover** the depth-eroded concentration subspace toward the mid-layer (0.866). So at
+  adapter scale with anatomical metadata, FINO's lever moves the needle in the right direction but
+  does not solve the erosion. F1b would need full-backbone capacity AND lesion-relevant metadata.
+- **Complementary experiment, if a FINO checkpoint ships**: run the probe on a *full* FINO-adapted
+  backbone (more capacity than our adapter) — the open question G1c could only lower-bound.
 
 ## Reproduce
 All jobs in `jobs/` (PEP-723 uv scripts) run via `hf jobs uv run --flavor <t4-medium|t4-small|cpu>

jobs/g1c_fino_adapter_job.py

@@ -0,0 +1,193 @@
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+#   "torch", "torchvision", "numpy", "pillow", "scikit-learn", "scipy",
+#   "huggingface_hub>=0.34", "dinov3 @ git+https://github.com/facebookresearch/dinov3",
+# ]
+# ///
+"""G1c — FINO-style metadata-guided adaptation: the TRAINED counterpart to G1a. HF Job (GPU).
+
+FINO (Gardes et al. 2026, arXiv:2606.05107) adapts FMs label-free via metadata-guided SSL TRAINING;
+no adapted checkpoint is released, so we test the MECHANISM at adapter scale. G1a showed the eroded
+late-layer concentration subspace is NOT recoverable training-free. Here, with MedDINOv3 FROZEN, we
+train a small residual-MLP adapter g (NO lesion labels) at a late layer under three conditions:
+  RAW       : no adapter (baseline late-layer localizability)
+  SSL_only  : augmentation-invariance objective only (the self-distillation pressure that globalizes)
+  FINO_style: SSL + METADATA guidance -- keep slice-position (continuous) and scan-id (discrete)
+              linearly decodable from pooled g(z) (preserve informative factors)
+Probe: lesion density-AUROC on g(z). FINO-direction supported if FINO_style > SSL_only and recovers
+toward the mid-layer target -> metadata-guided ADAPTATION succeeds where training-free steering
+failed, de-risking F1b. Honest proxy: adapter-scale, anatomical metadata; not a FINO reproduction.
+Labels eval-only. Emits G1C_RESULT.
+"""
+from __future__ import annotations
+import json, os, sys, time
+from pathlib import Path
+import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
+from PIL import Image
+from scipy import stats
+from sklearn.neighbors import NearestNeighbors
+from huggingface_hub import hf_hub_download
+sys.path.insert(0,"/mnt/processed/covtoken_code")
+from dinov3.models.vision_transformer import vit_base          # noqa: E402
+
+BACKBONE_REPO="ricklisz123/MedDINOv3-ViTB-16-CT-3M"; MNT=Path("/mnt")
+RAW_LIDC=MNT/"raw"/"lidc"; MASK_ROOT=MNT/"processed"/"lidc_v2"; OUT=MNT/"processed"/"covtoken"
+N_PATCH,CLS_OFF,GRID=196,5,14
+LAYERS=[int(x) for x in os.environ.get("LAYERS","8,11").split(",")]    # blocks 9, 12 (eroded)
+MID_TARGET=0.866                                                       # block-3 multi-seed peak
+BANK_SLICES=int(os.environ.get("BANK_SLICES","700")); EVAL_SLICES=int(os.environ.get("EVAL_SLICES","600"))
+STEPS=int(os.environ.get("STEPS","400")); LMETA=float(os.environ.get("LMETA","1.0"))
+CT_MEAN=np.array([0.485,0.456,0.406],np.float32); CT_STD=np.array([0.229,0.224,0.225],np.float32)
+def log(m): print(f"[g1c] {m}", flush=True)
+
+
+def load_backbone(device):
+    ck=hf_hub_download(BACKBONE_REPO,"model.pth",token=os.environ.get("HF_TOKEN"))
+    m=vit_base(drop_path_rate=0.0,layerscale_init=1e-5,n_storage_tokens=4,qkv_bias=False,mask_k_bias=True)
+    raw=torch.load(ck,map_location="cpu"); sd=raw.get("teacher",raw)
+    sd={(k[9:] if k.startswith("backbone.") else k):v for k,v in sd.items()}
+    m.load_state_dict(sd,strict=False); m.eval().to(device)
+    for p in m.parameters(): p.requires_grad_(False)
+    feats={}
+    for i,blk in enumerate(m.blocks):
+        def mk(i):
+            def h(_m,_i,out):
+                while isinstance(out,(list,tuple)): out=out[0]
+                feats[i]=out.detach()
+            return h
+        blk.register_forward_hook(mk(i))
+    return m,feats
+
+
+def to_t(pil):
+    a=(np.asarray(pil.resize((224,224),Image.BILINEAR),np.float32)/255.0-CT_MEAN)/CT_STD
+    return torch.from_numpy(a).permute(2,0,1)
+
+
+@torch.inference_mode()
+def feats_at(model,feats,imgs,device):
+    model.forward_features(imgs.to(device,torch.float32))
+    return {L:feats[L][:,CLS_OFF:CLS_OFF+N_PATCH,:].float() for L in LAYERS}   # GPU tensors
+
+
+class Adapter(nn.Module):
+    def __init__(self,d=768,h=256):
+        super().__init__(); self.n=nn.LayerNorm(d); self.a=nn.Linear(d,h); self.b=nn.Linear(h,d)
+        nn.init.zeros_(self.b.weight); nn.init.zeros_(self.b.bias)
+    def forward(self,z): return z + self.b(F.gelu(self.a(self.n(z))))
+
+
+def auroc(s,y):
+    s=np.asarray(s,float); y=np.asarray(y,int); pos,neg=y.sum(),len(y)-y.sum()
+    if pos==0 or neg==0: return float("nan")
+    r=stats.rankdata(s); return float((r[y==1].sum()-pos*(pos+1)/2)/(pos*neg))
+
+
+def boot_ci(s,y,n=1000,seed=0):
+    rng=np.random.default_rng(seed); s=np.asarray(s); y=np.asarray(y); idx=np.arange(len(y)); out=[]
+    for _ in range(n):
+        b=rng.choice(idx,len(idx),replace=True); out.append(auroc(s[b],y[b]))
+    return float(np.nanpercentile(out,2.5)),float(np.nanpercentile(out,97.5))
+
+
+def main():
+    t0=time.time(); device=torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    model,feats=load_backbone(device); rng=np.random.default_rng(0)
+    scan_split=json.load(open(hf_hub_download("Chucks90/eryon-data-pipelines","manifests/lidc/splits_v1.0.0.json",repo_type="dataset",token=os.environ.get("HF_TOKEN"))))["splits"]
+    # train images with metadata: zpos (continuous), scanid (discrete) -- NO lesion labels
+    scans=[];
+    for b in sorted(RAW_LIDC.glob("batch_*")):
+        for sd in b.iterdir():
+            if sd.is_dir() and scan_split.get(sd.name)=="train": scans.append(sd)
+    train=[]  # (path, zpos, scanid)
+    for sid,sd in enumerate(scans):
+        sl=sorted(sd.glob("slice_*.png")); n=len(sl)
+        for j,p in enumerate(sl): train.append((p, j/max(1,n-1), sid))
+    sel=rng.choice(len(train),min(BANK_SLICES,len(train)),replace=False); train=[train[i] for i in sel]
+    n_scans=len(scans)
+    ev=[]
+    for cd in sorted((MASK_ROOT/"test").iterdir()):
+        npz=cd/"patch_masks.npz"
+        if cd.is_dir() and npz.exists():
+            pm=np.load(npz)["patch_masks"]
+            for idx in range(len(pm)): ev.append((cd/f"slice_{idx:04d}.png", pm[idx]))
+    ev=[ev[i] for i in rng.choice(len(ev),min(EVAL_SLICES,len(ev)),replace=False)]
+    log(f"device={device.type}; train={len(train)} ({n_scans} scans) eval={len(ev)} layers={[L+1 for L in LAYERS]} steps={STEPS}")
+
+    # precompute frozen features once: orig + intensity-jitter + hflip, per layer; + metadata
+    flip_idx=torch.tensor((np.arange(N_PATCH).reshape(GRID,GRID)[:,::-1]).reshape(-1).copy(),device=device)
+    Zo={L:[] for L in LAYERS}; Zj={L:[] for L in LAYERS}; Zf={L:[] for L in LAYERS}; ZP=[]; ID=[]
+    for i in range(0,len(train),48):
+        ch=train[i:i+48]; pil=[Image.open(p).convert("RGB") for p,_,_ in ch]
+        o=feats_at(model,feats,torch.stack([to_t(im) for im in pil]),device)
+        jit=[Image.fromarray(np.clip(np.asarray(im,np.float32)*rng.uniform(0.85,1.15)+rng.uniform(-12,12),0,255).astype(np.uint8)) for im in pil]
+        j=feats_at(model,feats,torch.stack([to_t(im) for im in jit]),device)
+        fl=feats_at(model,feats,torch.stack([to_t(im.transpose(Image.FLIP_LEFT_RIGHT)) for im in pil]),device)
+        for L in LAYERS: Zo[L].append(o[L].cpu()); Zj[L].append(j[L].cpu()); Zf[L].append(fl[L].cpu())
+        ZP+=[z for _,z,_ in ch]; ID+=[s for _,_,s in ch]
+    Zo={L:torch.cat(v).to(device) for L,v in Zo.items()}; Zj={L:torch.cat(v).to(device) for L,v in Zj.items()}; Zf={L:torch.cat(v).to(device) for L,v in Zf.items()}
+    zpos=torch.tensor(ZP,dtype=torch.float32,device=device); sid=torch.tensor(ID,dtype=torch.long,device=device)
+    log(f"precomputed features in {time.time()-t0:.0f}s")
+    # eval features (orig only) per layer + labels
+    Ze={L:[] for L in LAYERS}; lab=[]
+    for i in range(0,len(ev),48):
+        ch=ev[i:i+48]; o=feats_at(model,feats,torch.stack([to_t(Image.open(p).convert("RGB")) for p,_ in ch]),device)
+        for L in LAYERS: Ze[L].append(o[L].cpu())
+        lab.append(np.stack([pm for _,pm in ch]).reshape(-1))
+    Ze={L:torch.cat(v).to(device) for L,v in Ze.items()}; lab=np.concatenate(lab)
+
+    def train_adapter(L, use_meta):
+        g=Adapter().to(device); pos_h=nn.Linear(768,1).to(device); id_h=nn.Linear(768,n_scans).to(device)
+        params=list(g.parameters())+(list(pos_h.parameters())+list(id_h.parameters()) if use_meta else [])
+        opt=torch.optim.Adam(params,lr=2e-3,weight_decay=1e-4); N=Zo[L].shape[0]
+        for step in range(STEPS):
+            bi=torch.randint(0,N,(64,),device=device)
+            zo=g(Zo[L][bi]); zj=g(Zj[L][bi]); zf=g(Zf[L][bi])
+            # SSL: invariance to jitter (same positions) + equivariance to flip (mirrored positions)
+            l_ssl=(1-F.cosine_similarity(zo,zj,dim=2)).mean() + (1-F.cosine_similarity(zo,zf[:,flip_idx,:],dim=2)).mean()
+            loss=l_ssl
+            if use_meta:
+                pool=zo.mean(1)
+                l_meta=F.mse_loss(pos_h(pool).squeeze(-1),zpos[bi]) + F.cross_entropy(id_h(pool),sid[bi])
+                loss=l_ssl+LMETA*l_meta
+            opt.zero_grad(); loss.backward(); opt.step()
+        g.eval(); return g
+
+    def probe(L, g):
+        with torch.no_grad():
+            ref=(g(Zo[L]) if g else Zo[L]).reshape(-1,768).cpu().numpy()
+            ref=ref[rng.choice(len(ref),min(50000,len(ref)),replace=False)]
+            nn_=NearestNeighbors(n_neighbors=11).fit(ref)
+            ze=(g(Ze[L]) if g else Ze[L]).reshape(-1,768).cpu().numpy()
+        d,_=nn_.kneighbors(ze); sc=d[:,1:].mean(1)
+        return auroc(sc,lab), boot_ci(sc,lab)
+
+    res={"backbone":"MedDINOv3","mid_layer_target":MID_TARGET,"steps":STEPS,"lambda_meta":LMETA,"by_layer":{}}
+    for L in LAYERS:
+        raw_a,raw_ci=probe(L,None)
+        g_ssl=train_adapter(L,use_meta=False); ssl_a,ssl_ci=probe(L,g_ssl)
+        g_fino=train_adapter(L,use_meta=True); fin_a,fin_ci=probe(L,g_fino)
+        row={"raw":{"auroc":round(raw_a,4),"ci":[round(x,4) for x in raw_ci]},
+             "ssl_only":{"auroc":round(ssl_a,4),"ci":[round(x,4) for x in ssl_ci]},
+             "fino_style":{"auroc":round(fin_a,4),"ci":[round(x,4) for x in fin_ci]},
+             "fino_minus_ssl":round(fin_a-ssl_a,4),"fino_minus_raw":round(fin_a-raw_a,4)}
+        res["by_layer"][str(L+1)]=row
+        log(f"  block {L+1}: raw {raw_a:.4f} | ssl_only {ssl_a:.4f} | FINO {fin_a:.4f} | FINO-ssl {fin_a-ssl_a:+.4f} | FINO-raw {fin_a-raw_a:+.4f}")
+
+    # verdict
+    deltas=[res["by_layer"][str(L+1)]["fino_minus_ssl"] for L in LAYERS]
+    fino_beats_ssl=all(res["by_layer"][str(L+1)]["fino_style"]["ci"][0] > res["by_layer"][str(L+1)]["ssl_only"]["auroc"] for L in LAYERS)
+    res["verdict"]={
+        "fino_beats_ssl_all_layers_ci": bool(fino_beats_ssl),
+        "mean_fino_minus_ssl": round(float(np.mean(deltas)),4),
+        "recovers_toward_mid": bool(any(res["by_layer"][str(L+1)]["fino_style"]["auroc"]>=res["by_layer"][str(L+1)]["raw"]["auroc"]+0.03 for L in LAYERS)),
+        "interpretation": ("FINO-style metadata-guided adaptation beats pure-SSL adaptation"
+            if np.mean(deltas)>0 else "metadata guidance does NOT beat SSL-only at adapter scale")
+            + f" (mean FINO-ssl {np.mean(deltas):+.4f}); contrast with G1a (training-free steering, no recovery)."}
+    res["elapsed_s"]=round(time.time()-t0,1)
+    OUT.mkdir(parents=True,exist_ok=True); (OUT/"g1c_fino_adapter.json").write_text(json.dumps(res,indent=2))
+    print("G1C_RESULT "+json.dumps(res),flush=True)
+
+
+if __name__=="__main__": main()

research_specs/RESEARCH_SPEC_v4.md

@@ -26,6 +26,23 @@ like mid. A deployable steering module for heads forced to use late features.
 
 ---
 
+## G1c — FINO-style metadata-guided adaptation (the trained counterpart to G1a)  ★ built
+
+**Context.** FINO (Gardès et al. 2026, arXiv:2606.05107) adapts FMs label-free via metadata-guided
+SSL *training*. No adapted checkpoint is released (FINO branch = code/recipes only), so full FINO is
+a pretraining run, not a job. We test the *mechanism* at adapter scale: G1a showed the eroded late-
+layer subspace is NOT recoverable training-free. Does a TRAINED, label-free, metadata-guided adapter
+recover it — and does FINO's metadata lever beat pure-SSL adaptation?
+
+**G1c.** Freeze MedDINOv3. At a late layer, train a small residual-MLP adapter g_theta (no lesion
+labels) under three conditions: (RAW) none; (SSL-only) augmentation-invariance objective — the pure
+self-distillation pressure that globalizes; (FINO-style) SSL + metadata guidance (keep slice-
+position [continuous] and scan-id [discrete] linearly decodable from pooled g(z) — preserve
+informative factors). Probe lesion density-AUROC on g(z). PASS-for-FINO-direction if FINO-style >
+SSL-only (CI excludes 0) and recovers toward the mid-layer target -> metadata-guided adaptation
+succeeds where training-free steering failed, de-risking F1b. Honest proxy: adapter-scale, anatomical
+metadata; not a FINO reproduction.
+
 ## G2 — Universality: the principle is not about lesions
 
 **Question.** Does the label-free concentration-subspace readout localize RARE structure in a

research_v4/g1c_fino_adapter.json

@@ -0,0 +1,26 @@
+{
+  "study": "G1c — FINO-style metadata-guided adaptation (trained counterpart to G1a)",
+  "status": "MEASURED / WEAK-NEGATIVE (directional mechanism signal; strong claim not supported)",
+  "context": "FINO (arXiv:2606.05107) adapts FMs label-free via metadata-guided SSL training; no checkpoint released, so tested at adapter scale. Question: does a TRAINED, label-free, metadata-guided adapter recover the eroded late-layer concentration subspace that training-free steering (G1a) could not?",
+  "mid_layer_target": 0.866,
+  "by_layer": {
+    "block9":  {"raw": 0.7145, "ssl_only": 0.6890, "fino_style": 0.7080, "fino_minus_ssl": 0.0190, "fino_minus_raw": -0.0065},
+    "block12": {"raw": 0.6513, "ssl_only": 0.6465, "fino_style": 0.6545, "fino_minus_ssl": 0.0079, "fino_minus_raw": 0.0032}
+  },
+  "verdict": {
+    "fino_beats_ssl_ci_significant": false,
+    "mean_fino_minus_ssl": 0.0135,
+    "recovers_toward_mid": false
+  },
+  "findings": [
+    "MECHANISM CONFIRMED DIRECTIONALLY: pure-SSL adaptation (augmentation-invariance only) HURT localizability at block 9 (0.715 -> 0.689) -- the globalization/invariance erosion, now reproduced by a TRAINED objective. FINO-style metadata guidance OFFSET this (+0.019 vs ssl), recovering back toward raw.",
+    "STRONG CLAIM NOT SUPPORTED: the FINO-vs-SSL gap is not CI-significant, and neither layer recovers toward the mid-layer concentration subspace (0.866). Metadata guidance at adapter scale mostly AVOIDS the SSL-induced loss rather than recovering the depth-eroded subspace.",
+    "CONSISTENT WITH G1a: the depth-erosion is robust even to a trained signal at adapter scale -- reinforcing that globalization is an entangling transformation, and that the deployment answer remains 'read the mid layer' (S2)."
+  ],
+  "honest_caveats": [
+    "Adapter-scale (residual MLP at one layer) is NOT full-backbone FINO pretraining; FINO adapts the whole backbone with far more capacity to reshape representations -- this is a lower bound on FINO's potential.",
+    "Anatomical metadata (slice-position, scan-id) captures anatomy, not pathology; FINO's premise needs metadata that captures the TASK-relevant factor. Lesion-relevant weak signal could do better."
+  ],
+  "implication_for_F1b": "NOT de-risked at this scale. F1b (concentration-preserving adaptation) would need (a) full-backbone capacity (not an adapter) AND (b) lesion-relevant metadata/weak supervision, not anatomical metadata. The bar for F1b is now better understood and higher than the FINO abstract suggested for rare-pathology localization specifically.",
+  "human_signoff": null
+}