Deploy InfiniSplat ZeroGPU demo

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/github_demo.gif filter=lfs diff=lfs merge=lfs -text
+examples/data/lidar_demo/eth3d_kicker.png filter=lfs diff=lfs merge=lfs -text
+examples/data/lidar_demo/eth3d_pipes.png filter=lfs diff=lfs merge=lfs -text
+examples/data/lidar_demo/waymo_147.png filter=lfs diff=lfs merge=lfs -text
+examples/data/lidar_demo/waymo_9.png filter=lfs diff=lfs merge=lfs -text
+examples/data/rgb_demo/eth3d_courtyard.png filter=lfs diff=lfs merge=lfs -text
+examples/data/rgb_demo/maksim-shutov-unsplash.jpg filter=lfs diff=lfs merge=lfs -text
+examples/data/rgb_demo/pexels-masi.jpg filter=lfs diff=lfs merge=lfs -text
+examples/data/rgb_demo/scannetpp_fe94fc30cf.JPG filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,42 @@
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.cache/
+*.egg-info/
+
+# Local environments and editors
+.venv/
+.env
+.vscode/
+.idea/
+.DS_Store
+
+# Runtime outputs
+/outputs/
+/wandb/
+/logs/
+/cache/
+/trash/
+/.trash/
+
+# Large model/data artifacts
+checkpoints/*.ckpt
+checkpoints/*.pth
+checkpoints/*.pt
+checkpoints/**/*.ckpt
+checkpoints/**/*.pth
+checkpoints/**/*.pt
+/data/
+/archives/
+
+# Demo outputs, but keep demo inputs under examples/data tracked when added.
+examples/data/**/outputs/
+*.ply
+*.mp4
+*.avi
+*.mov
+*.webm
+*.mkv
+*.compressed.ply
+*.sog

INSTALL.md

@@ -0,0 +1,72 @@
+## 📦 Environment & Checkpoints
+
+### 1) Create environment ([miniforge](https://github.com/conda-forge/miniforge) is recommended)
+
+If using `conda`, replace `mamba` with `conda` in the following commands and use the `conda-forge` channel when installing `gxx`:
+
+```bash
+mamba create -n infinisplat python=3.10
+mamba activate infinisplat
+
+# Optional: When gsplat compilation fails due to g++ version or CUDA toolkit issues.
+# mamba install gxx=10
+# mamba install nvidia/label/cuda-12.8.0::cuda-toolkit -c nvidia/label/cuda-12.8.0
+# export CUDA_HOME=$CONDA_PREFIX
+```
+
+### 2) Install dependencies
+
+```bash
+# Install uv
+pip install uv
+
+# Install PyTorch with CUDA 12.8
+uv pip install torch==2.9.0 torchvision==0.24.0 xformers==0.0.33.post1 --index-url https://download.pytorch.org/whl/cu128
+
+# Install package dependencies
+uv pip install -r requirements.txt
+```
+
+### 3) Optional output dependencies
+
+PLY export works without the dependencies in this section.
+
+Install `gsplat` only when novel-view video rendering is needed:
+
+```bash
+uv pip install git+https://github.com/nerfstudio-project/gsplat.git
+```
+
+Interactive HTML export requires Node.js and the PlayCanvas `splat-transform` CLI, but does not require `gsplat`:
+
+```bash
+npm install -g @playcanvas/splat-transform
+splat-transform -v
+```
+
+Missing optional dependencies are handled independently: without `gsplat`, MP4 rendering is skipped; without `splat-transform`, HTML conversion is skipped. In both cases, inference still exports the Gaussian PLY.
+
+### 4) Download checkpoints
+
+Download both released checkpoints with:
+
+```bash
+bash scripts/download_checkpoints.sh
+```
+
+This downloads the model files from [`PLUS-WAVE/InfiniSplat`](https://huggingface.co/PLUS-WAVE/InfiniSplat) into the local `checkpoints/` directory.
+
+#### Model Zoo
+
+| Category | Model | Use Case | Download |
+|---|---|---|---|
+| 3DGS | `InfiniSplat RGB` | RGB-Only Gaussian Inference | [infinisplat_rgb.ckpt](https://huggingface.co/PLUS-WAVE/InfiniSplat/blob/main/checkpoints/infinisplat_rgb.ckpt) |
+| 3DGS | `InfiniSplat Depth Sensor` | Gaussian Inference with RGB + Depth | [infinisplat_lidar.ckpt](https://huggingface.co/PLUS-WAVE/InfiniSplat/blob/main/checkpoints/infinisplat_lidar.ckpt) |
+
+The expected layout is:
+
+```text
+checkpoints/
+├── infinisplat_rgb.ckpt
+└── infinisplat_lidar.ckpt
+```

README.md

@@ -1,13 +1,144 @@
 ---
 title: InfiniSplat
-emoji: 💻
-colorFrom: green
-colorTo: indigo
+emoji: 🌌
+colorFrom: gray
+colorTo: green
 sdk: gradio
 sdk_version: 6.20.0
-python_version: '3.12'
+python_version: "3.10.13"
 app_file: app.py
-pinned: false
+short_description: Single-image Gaussian reconstruction
+startup_duration_timeout: 30m
+models:
+  - PLUS-WAVE/InfiniSplat
+preload_from_hub:
+  - PLUS-WAVE/InfiniSplat checkpoints/infinisplat_rgb.ckpt
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+<div align="center">
+
+<h1>🌌 InfiniSplat: Implicit Gaussian Decoding for Large-Baseline Monocular View Synthesis</h1>
+
+<div align="center">
+  <a href="https://pluswave.top/InfiniSplat-page/">
+    <img src="https://img.shields.io/badge/Project-Page-red?logo=googlechrome&logoColor=red">
+  </a>
+  <a href="#">
+    <img src="https://img.shields.io/badge/arXiv-Paper-blue?logo=arxiv&logoColor=blue">
+  </a>
+  <a href="https://pluswave.top/InfiniSplat-page/#visualization">
+    <img src="https://img.shields.io/badge/Gallery-Visualization-green?logo=googlephotos&logoColor=white">
+  </a>
+  <a href="https://huggingface.co/PLUS-WAVE/InfiniSplat">
+    <img src="https://img.shields.io/badge/HuggingFace-Model-yellow?logo=huggingface&logoColor=yellow">
+  </a>
+</div>
+
+<p align="center">
+  <a href="https://plus-wave.github.io/">Jiawei Wang*</a> •
+  <a href="https://ritianyu.github.io/">Hao Yu*</a> •
+  <a href="https://github.com/Edisoneh">Yongzhen Hu</a> •
+  <a href="https://github.com/shmily768">Xinyi Yang</a> •
+  <a href="#">Tao Ni</a> •
+  <a href="#">Xin Zhan</a> •
+  <a href="#">Junbo Chen</a> <br>
+  <a href="https://xzhou.me/">Xiaowei Zhou</a> •
+  <a href="https://csse.szu.edu.cn/staff/ruizhenhu/">Ruizhen Hu</a> •
+  <a href="https://pengsida.net/">Sida Peng</a>
+</p>
+
+<!-- <p align="center"><sup>* Equal contribution.</sup></p> -->
+
+</div>
+
+<div align="center">
+
+<img src="assets/github_demo.gif" alt="InfiniSplat Demo" width="90%" />
+
+</div>
+
+## 📣 News
+
+> **[2026-07]** 🎉 Inference code for RGB-only and depth-sensor-guided 3D Gaussian reconstruction is available now!
+
+## 🧩 What can InfiniSplat do?
+
+InfiniSplat supports two practical modes for single-image 3D Gaussian reconstruction:
+
+| Capability | Input | Output |
+| --- | --- | --- |
+| Monocular 3D Gaussian Reconstruction | RGB Image | 3DGS |
+| Depth-Sensor-Guided 3D Gaussian Reconstruction | RGB Image + Depth | 3DGS |
+
+## ⚙️ Installation
+
+Please see [INSTALL.md](INSTALL.md) for environment setup and checkpoint download.
+
+## 🚀 Inference
+
+### RGB Only
+
+Run a single image:
+
+```bash
+python -m src.demo.infer_batch_images --input examples/data/rgb_demo/pexels-masi.jpg
+```
+
+Run a directory using the bundled examples:
+
+```bash
+python -m src.demo.infer_batch_images --input examples/data/rgb_demo
+```
+
+### Depth-Sensor-Guided Reconstruction
+
+Run a single RGB and depth pair with matching filename stems in the same directory:
+
+```bash
+python -m src.demo.infer_batch_images \
+  --mode lidar \
+  --input examples/data/lidar_demo/eth3d_kicker.png
+```
+
+Run the bundled RGB and depth pairs:
+
+```bash
+python -m src.demo.infer_batch_images \
+  --mode lidar \
+  --input examples/data/lidar_demo
+```
+
+See [docs/inference.md](docs/inference.md) for camera parameters, recursive directory scanning, output control, and other optional arguments.
+
+## Hugging Face Space
+
+The root `app.py` exposes RGB reconstruction as a Gradio ZeroGPU Space. The encoder
+is loaded once at startup, GPU inference writes a temporary CPU artifact, and CPU
+post-processing exports a Gaussian PLY plus a standalone SuperSplat HTML viewer.
+The released RGB checkpoint is downloaded from `PLUS-WAVE/InfiniSplat`. This RGB
+path does not require xFormers or any custom CUDA extension.
+
+Run the Space app locally from the prepared inference environment:
+
+```bash
+pip install spaces
+INFINISPLAT_CHECKPOINT=checkpoints/infinisplat_rgb.ckpt python app.py
+```
+
+Select ZeroGPU in the Space hardware settings after creating the Gradio Space.
+
+## 🙏 Acknowledgments
+
+We sincerely thank the authors of [DINOv3](https://github.com/facebookresearch/dinov3), [Depth Pro](https://github.com/apple/ml-depth-pro), [InfiniDepth](https://github.com/zju3dv/InfiniDepth), and [gsplat](https://github.com/nerfstudio-project/gsplat) for their excellent work. InfiniSplat is built on top of these projects.
+
+---
+
+<div align="center">
+
+<img src="https://raw.githubusercontent.com/Tarikul-Islam-Anik/Animated-Fluent-Emojis/master/Emojis/Hand%20gestures/Folded%20Hands%20Light%20Skin%20Tone.png" alt="Thanks" width="25" height="25" />
+
+**Thank you for your interest in InfiniSplat!**
+
+<sub>⭐ Star this repo if you find it interesting!</sub>
+
+</div>

app.py

@@ -0,0 +1,28 @@
+from __future__ import annotations
+
+import os
+
+os.environ.setdefault("GRADIO_SSR_MODE", "false")
+
+import spaces  # noqa: F401 - must patch torch before model modules are imported
+
+from src.demo.hf_runtime import InfiniSplatRuntime
+from src.demo.hf_ui import APP_CSS, APP_THEME, OUTPUT_ROOT, create_demo
+
+
+runtime = InfiniSplatRuntime.load()
+demo = create_demo(runtime)
+
+
+if __name__ == "__main__":
+    demo.launch(
+        server_name="0.0.0.0",
+        server_port=int(os.environ.get("PORT", "7860")),
+        allowed_paths=[str(OUTPUT_ROOT)],
+        max_file_size="20mb",
+        show_error=True,
+        ssr_mode=False,
+        footer_links=[],
+        theme=APP_THEME,
+        css=APP_CSS,
+    )

config/experiment/infinisplat_hypersim_lidar.yaml

@@ -0,0 +1,5 @@
+# @package _global_
+defaults:
+  - override /model/encoder: infinisplat_infinidepth
+  - override /model/decoder: gsplat
+  - _self_

config/experiment/infinisplat_hypersim_rgb.yaml

@@ -0,0 +1,5 @@
+# @package _global_
+defaults:
+  - override /model/encoder: infinisplat
+  - override /model/decoder: gsplat
+  - _self_

config/inference.yaml

@@ -0,0 +1,4 @@
+defaults:
+  - model/encoder: infinisplat
+  - model/decoder: gsplat
+  - _self_

config/model/decoder/gsplat.yaml

@@ -0,0 +1,10 @@
+# ============================================================================
+# GSplat Decoder Configuration
+# ============================================================================
+# This is the SINGLE SOURCE OF TRUTH for all default configuration values.
+# The corresponding Python dataclass (DecoderGsplatCfg) only provides
+# type annotations without default values.
+# ============================================================================
+
+name: gsplat
+background_color: [0.0, 0.0, 0.0]

config/model/encoder/infinisplat.yaml

@@ -0,0 +1,28 @@
+name: infinisplat
+
+sample_point_num: 1500000
+
+image_basic_dim: 128
+
+image_backbone_type: vitl16
+
+implicit_gs_query_batch_size: 80000
+implicit_gs_hidden_list: [1024, 256, 64]
+
+gaussian_decoder:
+  delta_factor_xy: 0.001
+  delta_factor_z: 0.001
+  delta_factor_scale: 1.0
+  delta_factor_rotation: 1.0
+  delta_factor_color: 0.1
+  delta_factor_opacity: 1.0
+  scale_min: 0.0
+  scale_max: 4.0
+  init_opacity: 0.5
+  opacity_min: 0.01
+  opacity_max: 0.99
+  rgb_min: 0.01
+  rgb_max: 0.99
+  depth_normalization_min: 1.0
+  depth_normalization_max: 100.0
+  base_scale_multiplier: 1.0

config/model/encoder/infinisplat_infinidepth.yaml

@@ -0,0 +1,28 @@
+name: infinisplat_infinidepth
+
+sample_point_num: 1500000
+
+image_basic_dim: 128
+
+image_backbone_type: vitl16
+
+implicit_gs_query_batch_size: 80000
+implicit_gs_hidden_list: [1024, 256, 64]
+
+gaussian_decoder:
+  delta_factor_xy: 0.001
+  delta_factor_z: 0.001
+  delta_factor_scale: 1.0
+  delta_factor_rotation: 1.0
+  delta_factor_color: 0.1
+  delta_factor_opacity: 1.0
+  scale_min: 0.0
+  scale_max: 4.0
+  init_opacity: 0.5
+  opacity_min: 0.01
+  opacity_max: 0.99
+  rgb_min: 0.01
+  rgb_max: 0.99
+  depth_normalization_min: 1.0
+  depth_normalization_max: 100.0
+  base_scale_multiplier: 1.0

config/viewer_settings.json

@@ -0,0 +1,49 @@
+{
+  "version": 2,
+  "tonemapping": "none",
+  "highPrecisionRendering": false,
+  "background": {
+    "color": [0.0, 0.0, 0.0]
+  },
+  "postEffectSettings": {
+    "sharpness": {
+      "enabled": false,
+      "amount": 0
+    },
+    "bloom": {
+      "enabled": false,
+      "intensity": 1,
+      "blurLevel": 2
+    },
+    "grading": {
+      "enabled": false,
+      "brightness": 0,
+      "contrast": 1,
+      "saturation": 1,
+      "tint": [1, 1, 1]
+    },
+    "vignette": {
+      "enabled": false,
+      "intensity": 0.5,
+      "inner": 0.3,
+      "outer": 0.75,
+      "curvature": 1
+    },
+    "fringing": {
+      "enabled": false,
+      "intensity": 0.5
+    }
+  },
+  "animTracks": [],
+  "cameras": [
+    {
+      "initial": {
+        "position": [0.0, 0.0, 0.0],
+        "target": [0.0, 0.0, 1.0],
+        "fov": 60.0
+      }
+    }
+  ],
+  "annotations": [],
+  "startMode": "default"
+}

docs/inference.md

@@ -0,0 +1,154 @@
+# Inference Guide
+
+Run all commands from the repository root through the Python entry point:
+
+```bash
+python -m src.demo.infer_batch_images [OPTIONS]
+```
+
+## Modes
+
+| Mode | Encoder configuration | Default checkpoint | Prompt depth |
+| --- | --- | --- | --- |
+| `rgb` | RGB-only | `checkpoints/infinisplat_rgb.ckpt` | Not used |
+| `lidar` | RGB + prompt-conditioned InfiniDepth | `checkpoints/infinisplat_lidar.ckpt` | Required |
+
+The selected checkpoint contains the full inference model. Encoder and decoder weights are not downloaded separately.
+
+## Input discovery
+
+`--input` accepts one image or a directory. Supported image extensions are `.jpg`, `.jpeg`, `.png`, `.bmp`, and `.webp`. Directory entries are sorted to make batch selection deterministic.
+
+By default, only the selected directory is scanned. Add `--recursive` to scan its subdirectories and `--limit N` to process only the first `N` selected images. A limit of `0` means no cap.
+
+When `--input` is omitted, each mode uses its bundled example directory:
+
+```text
+rgb   -> examples/data/rgb_demo
+lidar -> examples/data/lidar_demo
+```
+
+## Depth pairing
+
+By default, depth files are searched in the input directory and paired by filename stem. For a single-image input, this is the directory containing the image. For example:
+
+```text
+data/frame_001.jpg
+data/frame_001.npy
+```
+
+Run this pair with:
+
+```bash
+python -m src.demo.infer_batch_images \
+  --mode lidar \
+  --input data/frame_001.jpg
+```
+
+or to process all images and depths in a directory:
+
+```bash
+python -m src.demo.infer_batch_images \
+  --mode lidar \
+  --input data
+```
+
+---
+
+
+To explicitly select the depth file for a single image, use `--prompt-depth`:
+
+```bash
+python -m src.demo.infer_batch_images \
+  --mode lidar \
+  --input /path/to/frame_001.jpg \
+  --prompt-depth /path/to/frame_001.npy
+```
+
+To keep depth files in a separate directory, set `--prompt-depth-dir`; each image is still paired with a depth file that has the same stem:
+
+```bash
+python -m src.demo.infer_batch_images \
+  --mode lidar \
+  --input /path/to/images \
+  --prompt-depth-dir /path/to/depths
+```
+
+`--prompt-depth` applies the same file to every selected image, so it should normally be used only with a single-image input. Candidate extensions are tried in this order when pairing automatically: `.npz`, `.npy`, `.png`, `.h5`, `.hdf5`, `.exr`.
+
+## Depth input format
+
+Prompt depth must be spatially aligned with the RGB image and use larger values for farther points. Metric scale is optional: scale-ambiguous depth maps are also supported, while metric input preserves the scene scale in the exported 3DGS. Disparity or inverse depth must be converted to depth first.
+
+Depth arrays should use shape `[H, W]`. Plain arrays and dense maps are accepted; `.npz` files may instead store a sparse `mask` and `value` pair. Valid values must be finite and strictly between 1 and 100 after decoding, so relative depth in `[0, 1]` must be rescaled first. Integer PNG values above 255 are treated as millimeters and divided by 1000. At most 1500 valid samples are used as prompts.
+
+## Camera intrinsics
+
+Camera parameters are resolved in the following order:
+
+1. `--intrinsics-file`: a YAML or JSON file containing a 3 x 3 pixel-space matrix.
+2. `--focal-px`: focal length in pixels; `fx = fy`, with a centered principal point.
+3. `--focal-mm`: 35 mm full-frame-equivalent focal length.
+4. Image EXIF focal length.
+5. A fixed 30 mm full-frame-equivalent fallback.
+
+The three command-line overrides are mutually exclusive. One override is reused for every image in a batch.
+
+An intrinsics file may place the matrix at either `intrinsics_px` or `camera.intrinsics_px`:
+
+```yaml
+intrinsics_px:
+  - [1200.0, 0.0, 768.0]
+  - [0.0, 1200.0, 576.0]
+  - [0.0, 0.0, 1.0]
+```
+
+Values must describe the original input image in pixels. The loader scales the matrix automatically during preprocessing.
+
+## Optional video render resolution
+
+When the optional `gsplat` package is installed, videos render at the original image resolution when possible. Very large frames are scaled down to a maximum long edge of 3840 pixels and a maximum area of `3840 x 2160` pixels. Each video contains 60 frames at 10 FPS.
+
+## Outputs and resume behavior
+
+The default output root is `outputs/demo/<mode>`. Every image receives its own directory:
+
+```text
+outputs/demo/rgb/example/
+├── example.ply
+├── example.mp4   # optional: requires gsplat
+└── example.html  # optional: requires splat-transform
+```
+
+PLY export is always enabled and does not require `gsplat`. Video export is enabled by default when `gsplat` is installed; otherwise it is skipped with a warning. Pass `--no-video` to disable it explicitly. HTML export is independent of `gsplat` and is skipped with a warning when the optional `splat-transform` executable is unavailable.
+
+If every requested artifact already exists, the case is skipped. If the PLY and requested video exist but HTML is missing, only HTML conversion runs. Pass `--overwrite` to recompute requested outputs.
+
+The HTML converter reads `config/viewer_settings.json` directly. It does not generate a hidden viewer-settings file.
+
+## Common options
+
+| Option | Description |
+| --- | --- |
+| `--mode {rgb,lidar}` | Select the model mode. |
+| `--checkpoint PATH` | Override the checkpoint selected by the mode. |
+| `--input PATH` | Process one image or a directory. |
+| `--output-dir PATH` | Override `outputs/demo/<mode>`. |
+| `--recursive` | Search input subdirectories. |
+| `--limit N` | Process at most `N` selected images; `0` means all. |
+| `--overwrite` | Recompute outputs that already exist. |
+| `--device DEVICE` | Override automatic device selection, for example `cuda:0`. |
+| `--intrinsics-file PATH` | Use a 3 x 3 pixel-space intrinsics matrix. |
+| `--focal-px VALUE` | Use a focal length in pixels. |
+| `--focal-mm VALUE` | Use a full-frame-equivalent focal length in millimeters. |
+| `--prompt-depth PATH` | Use one prompt-depth file. |
+| `--prompt-depth-dir PATH` | Pair depth files by image stem. |
+| `--disable-floater-filter` | Keep Gaussians removed by the final floater filter. |
+| `--no-video` | Skip MP4 rendering. |
+| `--no-export-html` | Skip HTML viewer export. |
+
+Print the authoritative option list with:
+
+```bash
+python -m src.demo.infer_batch_images --help
+```

examples/data/lidar_demo/eth3d_kicker.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d65d75094be01eb8bab4f99ca859c94b3d04f2c0a68884c470385feb06fbb165
+size 632654

examples/data/lidar_demo/eth3d_pipes.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e22a52079f426bb0373e5902a5f77da50fd0426d0ee9f134a64635bd79278597
+size 677470

examples/data/lidar_demo/waymo_147.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:befe9ad8a84cbd082927486d276b4a25db94247a67e8e0ea11def64b9a6c627b
+size 84094

examples/data/lidar_demo/waymo_9.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b750d2180e31896b1363fa6edbff735af9d053fcceb5c31e5d8c55211aa5401c
+size 91954

packages.txt

@@ -0,0 +1,6 @@
+libgl1
+libglib2.0-0
+libvulkan1
+mesa-vulkan-drivers
+nodejs
+npm

requirements.txt

@@ -0,0 +1,30 @@
+# Keep the torch family aligned with the supported ZeroGPU runtime.
+torch==2.9.1
+torchvision==0.24.1
+
+# Inference configuration and CLI
+hydra-core
+omegaconf
+dacite
+termcolor
+rich
+tqdm
+
+# Tensor, vision, and geometry
+numpy<2.0
+scipy
+pandas
+h5py
+einops>=0.4.1
+jaxtyping
+opencv-python-headless
+scikit-learn
+Pillow
+imageio[ffmpeg]
+matplotlib
+plyfile
+torchmetrics
+
+# Model backbones and checkpoint download
+timm
+regex

scripts/download_checkpoints.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")/.."
+
+python - <<'PY'
+from huggingface_hub import snapshot_download
+
+snapshot_download(
+    repo_id="PLUS-WAVE/InfiniSplat",
+    allow_patterns="checkpoints/*.ckpt",
+    local_dir=".",
+)
+PY

src/__init__.py

@@ -0,0 +1 @@
+"""InfiniSplat source package."""

src/demo/__init__.py

@@ -0,0 +1,2 @@
+"""Single-image demo inference utilities."""
+

src/demo/config.py

@@ -0,0 +1,32 @@
+from dataclasses import dataclass
+from typing import Type, TypeVar
+
+from dacite import from_dict
+from omegaconf import DictConfig, OmegaConf
+
+from src.model.decoder import DecoderCfg
+from src.model.encoder import EncoderCfg
+
+
+@dataclass
+class ModelCfg:
+    decoder: DecoderCfg
+    encoder: EncoderCfg
+
+
+@dataclass
+class RootCfg:
+    model: ModelCfg
+
+
+T = TypeVar("T")
+
+
+def load_typed_config(cfg: DictConfig, data_class: Type[T]) -> T:
+    """Convert one resolved Hydra config into its inference dataclass."""
+    return from_dict(data_class, OmegaConf.to_container(cfg, resolve=True))
+
+
+def load_typed_root_config(cfg: DictConfig) -> RootCfg:
+    """Load the typed root configuration used by demo inference."""
+    return load_typed_config(cfg, RootCfg)

src/demo/hf_runtime.py

@@ -0,0 +1,320 @@
+from __future__ import annotations
+
+import base64
+import hashlib
+import json
+import os
+import shlex
+import shutil
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Sequence
+
+import torch
+from huggingface_hub import hf_hub_download
+
+from src.demo.infer_single_image import (
+    load_demo_config,
+    load_demo_encoder,
+    load_demo_image_bundle,
+    patch_supersplat_html_auto_rotate,
+    patch_supersplat_html_viewer_bridge,
+    run_single_image_inference,
+)
+from src.model.encoder import Encoder
+from src.utils.gaussians import Gaussians3D, save_ply
+
+MODEL_REPO_ID = "PLUS-WAVE/InfiniSplat"
+RGB_CHECKPOINT_FILE = "checkpoints/infinisplat_rgb.ckpt"
+RGB_EXPERIMENT = "infinisplat_hypersim_rgb"
+SPLAT_TRANSFORM_PACKAGE = "@playcanvas/splat-transform@2.1.0"
+VIEWER_SETTINGS = Path(__file__).resolve().parents[2] / "config" / "viewer_settings.json"
+ARTIFACT_VERSION = 1
+VIEWER_ASSET_FILENAMES = ("index.js", "index.css", "settings.json")
+
+
+@dataclass(frozen=True)
+class GaussianArtifact:
+    """CPU-resident Gaussian tensors and camera metadata for one request."""
+
+    gaussians: Gaussians3D
+    focal_length_px: float
+    image_shape: tuple[int, int]
+
+
+@dataclass(frozen=True)
+class ExportedArtifacts:
+    """Public output files generated for one request."""
+
+    scene_ply: Path
+    viewer_html: Path
+    standalone_html: Path
+
+
+def _gaussians_to_dict(gaussians: Gaussians3D) -> dict[str, torch.Tensor | None]:
+    return {
+        "mean_vectors": gaussians.mean_vectors.detach().cpu(),
+        "singular_values": gaussians.singular_values.detach().cpu(),
+        "quaternions": gaussians.quaternions.detach().cpu(),
+        "colors": gaussians.colors.detach().cpu(),
+        "opacities": gaussians.opacities.detach().cpu(),
+        "covariances": None if gaussians.covariances is None else gaussians.covariances.detach().cpu(),
+    }
+
+
+def save_gaussian_artifact(artifact: GaussianArtifact, path: Path) -> Path:
+    """Serialize one request artifact using the safe torch data subset."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    torch.save(
+        {
+            "version": ARTIFACT_VERSION,
+            "gaussians": _gaussians_to_dict(artifact.gaussians),
+            "focal_length_px": artifact.focal_length_px,
+            "image_shape": list(artifact.image_shape),
+        },
+        path,
+    )
+    return path
+
+
+def load_gaussian_artifact(path: Path) -> GaussianArtifact:
+    """Load an internal request artifact without permitting arbitrary objects."""
+    payload = torch.load(path, map_location="cpu", weights_only=True)
+    if payload["version"] != ARTIFACT_VERSION:
+        raise ValueError(f"Unsupported Gaussian artifact version: {payload['version']}")
+    tensors = payload["gaussians"]
+    gaussians = Gaussians3D(
+        mean_vectors=tensors["mean_vectors"],
+        singular_values=tensors["singular_values"],
+        quaternions=tensors["quaternions"],
+        colors=tensors["colors"],
+        opacities=tensors["opacities"],
+        covariances=tensors["covariances"],
+    )
+    return GaussianArtifact(
+        gaussians=gaussians,
+        focal_length_px=float(payload["focal_length_px"]),
+        image_shape=tuple(int(value) for value in payload["image_shape"]),
+    )
+
+
+def _default_splat_transform_prefix() -> list[str]:
+    override = os.environ.get("SPLAT_TRANSFORM")
+    if override:
+        return shlex.split(override)
+    return ["npx", "--yes", SPLAT_TRANSFORM_PACKAGE]
+
+
+def build_splat_transform_command(
+    scene_ply: Path,
+    output_html: Path,
+    viewer_settings: Path = VIEWER_SETTINGS,
+    command_prefix: Sequence[str] | None = None,
+) -> list[str]:
+    """Build the fixed, non-interactive SuperSplat HTML conversion command."""
+    prefix = list(command_prefix) if command_prefix is not None else _default_splat_transform_prefix()
+    return [
+        *prefix,
+        "--quiet",
+        "--overwrite",
+        "--unbundled",
+        "--viewer-settings",
+        str(viewer_settings),
+        str(scene_ply),
+        "--filter-harmonics",
+        "0",
+        str(output_html),
+    ]
+
+
+def _replace_once(source: str, old: str, new: str, description: str) -> str:
+    if source.count(old) != 1:
+        raise RuntimeError(
+            f"Could not bundle SuperSplat {description}; the viewer template changed."
+        )
+    return source.replace(old, new, 1)
+
+
+def build_standalone_viewer(viewer_html: Path, output_html: Path) -> Path:
+    """Bundle an unbundled SuperSplat viewer for single-file download."""
+    output_dir = viewer_html.parent
+    scene_sog = viewer_html.with_suffix(".sog")
+    source = viewer_html.read_text(encoding="utf-8")
+    css = (output_dir / "index.css").read_text(encoding="utf-8")
+    javascript = (output_dir / "index.js").read_text(encoding="utf-8")
+    settings = json.loads((output_dir / "settings.json").read_text(encoding="utf-8"))
+    encoded_scene = base64.b64encode(scene_sog.read_bytes()).decode("ascii")
+
+    source = _replace_once(
+        source,
+        '<link rel="stylesheet" href="./index.css">',
+        f"<style>\n{css}\n        </style>",
+        "stylesheet",
+    )
+    source = _replace_once(
+        source,
+        "import { main } from './index.js';",
+        javascript,
+        "script",
+    )
+    source = _replace_once(
+        source,
+        "settings: fetch(settingsUrl).then(response => response.json())",
+        f"settings: {json.dumps(settings, separators=(',', ':'), ensure_ascii=False)}",
+        "settings",
+    )
+    source = _replace_once(
+        source,
+        f'fetch("{scene_sog.name}")',
+        f'fetch("data:application/octet-stream;base64,{encoded_scene}")',
+        "scene data",
+    )
+    output_html.write_text(source, encoding="utf-8")
+    return output_html
+
+
+def install_shared_viewer_assets(viewer_html: Path) -> Path:
+    """Point an unbundled viewer at content-addressed shared static assets."""
+    asset_paths = [viewer_html.parent / name for name in VIEWER_ASSET_FILENAMES]
+    hasher = hashlib.sha256()
+    for asset_path in asset_paths:
+        hasher.update(asset_path.name.encode("utf-8"))
+        hasher.update(asset_path.read_bytes())
+    digest = hasher.hexdigest()[:16]
+    shared_dir = viewer_html.parent.parent / "_viewer_assets" / digest
+    shared_dir.mkdir(parents=True, exist_ok=True)
+    for asset_path in asset_paths:
+        destination = shared_dir / asset_path.name
+        if not destination.exists():
+            shutil.copyfile(asset_path, destination)
+
+    shared_prefix = f"../_viewer_assets/{digest}"
+    source = viewer_html.read_text(encoding="utf-8")
+    source = _replace_once(
+        source,
+        "./index.css",
+        f"{shared_prefix}/index.css",
+        "shared stylesheet path",
+    )
+    source = _replace_once(
+        source,
+        "./index.js",
+        f"{shared_prefix}/index.js",
+        "shared script path",
+    )
+    source = _replace_once(
+        source,
+        "./settings.json",
+        f"{shared_prefix}/settings.json",
+        "shared settings path",
+    )
+    viewer_html.write_text(source, encoding="utf-8")
+    return shared_dir
+
+
+def export_gaussian_artifact(
+    artifact_path: Path,
+    output_dir: Path,
+    command_prefix: Sequence[str] | None = None,
+) -> ExportedArtifacts:
+    """Export unchanged Gaussians for a cached viewer and standalone download."""
+    artifact = load_gaussian_artifact(artifact_path)
+    gaussians = artifact.gaussians
+
+    output_dir.mkdir(parents=True, exist_ok=True)
+    scene_ply = output_dir / "scene.ply"
+    viewer_html = output_dir / "viewer.html"
+    standalone_html = output_dir / "scene.html"
+    save_ply(
+        gaussians=gaussians,
+        f_px=artifact.focal_length_px,
+        image_shape=artifact.image_shape,
+        path=scene_ply,
+    )
+    converter_environment = os.environ.copy()
+    xdg_runtime_dir = output_dir / ".xdg-runtime"
+    xdg_runtime_dir.mkdir(mode=0o700, exist_ok=True)
+    converter_environment["XDG_RUNTIME_DIR"] = str(xdg_runtime_dir)
+    subprocess.run(
+        build_splat_transform_command(
+            scene_ply=scene_ply,
+            output_html=viewer_html,
+            command_prefix=command_prefix,
+        ),
+        check=True,
+        env=converter_environment,
+    )
+    patch_supersplat_html_auto_rotate(viewer_html)
+    patch_supersplat_html_viewer_bridge(
+        viewer_html,
+        viewer_script_path=output_dir / "index.js",
+    )
+    build_standalone_viewer(viewer_html, standalone_html)
+    install_shared_viewer_assets(viewer_html)
+    return ExportedArtifacts(
+        scene_ply=scene_ply,
+        viewer_html=viewer_html,
+        standalone_html=standalone_html,
+    )
+
+
+def resolve_rgb_checkpoint() -> Path:
+    """Resolve the local override or download the released RGB checkpoint."""
+    local_checkpoint = os.environ.get("INFINISPLAT_CHECKPOINT")
+    if local_checkpoint:
+        checkpoint_path = Path(local_checkpoint)
+        if not checkpoint_path.is_file():
+            raise FileNotFoundError(f"Checkpoint not found: {checkpoint_path}")
+        return checkpoint_path
+
+    return Path(
+        hf_hub_download(
+            repo_id=os.environ.get("INFINISPLAT_MODEL_REPO", MODEL_REPO_ID),
+            filename=RGB_CHECKPOINT_FILE,
+            revision=os.environ.get("INFINISPLAT_MODEL_REVISION", "main"),
+        )
+    )
+
+
+class InfiniSplatRuntime:
+    """One process-wide RGB encoder reused by all web requests."""
+
+    def __init__(self, encoder: Encoder, device: torch.device) -> None:
+        self.encoder = encoder
+        self.device = device
+
+    @classmethod
+    def load(
+        cls,
+        checkpoint_path: Path | None = None,
+        device: str | torch.device | None = None,
+    ) -> "InfiniSplatRuntime":
+        resolved_device = torch.device(
+            device or ("cuda" if torch.cuda.is_available() else "cpu")
+        )
+        cfg = load_demo_config(RGB_EXPERIMENT)
+        encoder = load_demo_encoder(
+            cfg=cfg,
+            checkpoint_path=checkpoint_path or resolve_rgb_checkpoint(),
+            device=resolved_device,
+        )
+        return cls(encoder=encoder, device=resolved_device)
+
+    @torch.inference_mode()
+    def infer_to_artifact(self, image_path: Path, artifact_path: Path) -> Path:
+        """Run one RGB reconstruction and persist CPU tensors for post-processing."""
+        image_bundle = load_demo_image_bundle(image_path=image_path)
+        encoder_output = run_single_image_inference(
+            encoder=self.encoder,
+            image=image_bundle.inference_image,
+            intrinsics_px=image_bundle.inference_intrinsics.intrinsics_px,
+            device=self.device,
+        )
+        _, height, width = image_bundle.inference_image.shape
+        artifact = GaussianArtifact(
+            gaussians=encoder_output["gaussians"].to("cpu"),
+            focal_length_px=image_bundle.inference_intrinsics.focal_length_px,
+            image_shape=(height, width),
+        )
+        return save_gaussian_artifact(artifact, artifact_path)

src/demo/hf_ui.py

@@ -0,0 +1,658 @@
+from __future__ import annotations
+
+import html
+import uuid
+from pathlib import Path
+from urllib.parse import quote
+
+import spaces
+import gradio as gr
+from gradio.utils import get_upload_folder
+
+from src.demo.hf_runtime import InfiniSplatRuntime, export_gaussian_artifact
+
+OUTPUT_ROOT = Path(get_upload_folder()).resolve() / "infinisplat"
+OUTPUT_ROOT.mkdir(parents=True, exist_ok=True)
+_runtime: InfiniSplatRuntime | None = None
+RGB_EXAMPLES = [
+    ("examples/data/rgb_demo/eth3d_courtyard.png", "Courtyard"),
+    ("examples/data/rgb_demo/maksim-shutov-unsplash.jpg", "Unsplash"),
+    ("examples/data/rgb_demo/pexels-masi.jpg", "Pexels"),
+    ("examples/data/rgb_demo/scannetpp_fe94fc30cf.JPG", "ScanNet++"),
+]
+APP_THEME = gr.themes.Soft(
+    primary_hue="emerald",
+    secondary_hue="amber",
+    neutral_hue="zinc",
+    spacing_size="sm",
+    radius_size="sm",
+    font=("ui-sans-serif", "system-ui", "sans-serif"),
+).set(
+    body_background_fill="#f5f6f6",
+    body_background_fill_dark="#f5f6f6",
+    body_text_color="#17201e",
+    body_text_color_dark="#17201e",
+    body_text_color_subdued="#68716f",
+    body_text_color_subdued_dark="#68716f",
+    background_fill_primary="#ffffff",
+    background_fill_primary_dark="#ffffff",
+    background_fill_secondary="#f8f9f9",
+    background_fill_secondary_dark="#f8f9f9",
+    block_background_fill="#ffffff",
+    block_background_fill_dark="#ffffff",
+    block_label_text_color="#17201e",
+    block_label_text_color_dark="#17201e",
+    block_title_text_color="#17201e",
+    block_title_text_color_dark="#17201e",
+    input_background_fill="#ffffff",
+    input_background_fill_dark="#ffffff",
+    input_placeholder_color="#68716f",
+    input_placeholder_color_dark="#68716f",
+    button_primary_background_fill="#087f62",
+    button_primary_background_fill_dark="#087f62",
+    button_primary_text_color="#ffffff",
+    button_primary_text_color_dark="#ffffff",
+    button_secondary_background_fill="#f8f9f9",
+    button_secondary_background_fill_dark="#f8f9f9",
+    button_secondary_text_color="#17201e",
+    button_secondary_text_color_dark="#17201e",
+)
+APP_CSS = """
+:root {
+    color-scheme: light;
+    --app-bg: #f5f6f6;
+    --surface: #ffffff;
+    --surface-muted: #f8f9f9;
+    --border: #dfe3e1;
+    --border-strong: #c6ceca;
+    --text: #17201e;
+    --muted: #68716f;
+    --viewer: #101413;
+    --primary: #087f62;
+    --primary-hover: #066b53;
+    --accent: #c27616;
+}
+html,
+body {
+    background: var(--app-bg) !important;
+    color: var(--text) !important;
+}
+.gradio-container {
+    max-width: none !important;
+    min-height: 100vh;
+    padding: 0 !important;
+    background: var(--app-bg) !important;
+    color: var(--text);
+}
+.gradio-container *,
+.dark .gradio-container * {
+    letter-spacing: 0 !important;
+}
+#app-title {
+    padding: 28px 2px 18px;
+    background: transparent !important;
+}
+#app-title h1 {
+    margin: 0 !important;
+    color: var(--text);
+    font-size: clamp(1.75rem, 2.5vw, 2.2rem);
+    font-weight: 720;
+    line-height: 1;
+}
+#app-main {
+    width: calc(100% - clamp(28px, 6vw, 80px));
+    max-width: 1440px;
+    margin: 0 auto !important;
+    gap: 0 !important;
+}
+.section-heading {
+    display: flex;
+    align-items: baseline;
+    justify-content: space-between;
+    gap: 16px;
+}
+.section-heading h2 {
+    margin: 0;
+    color: var(--text);
+    font-size: 0.9rem;
+    font-weight: 700;
+}
+.section-heading span {
+    color: var(--muted);
+    font-size: 0.75rem;
+}
+#workspace {
+    margin: 0 0 26px !important;
+    gap: 18px;
+    align-items: stretch;
+}
+.tool-panel {
+    min-width: 0 !important;
+    overflow: hidden;
+    gap: 0 !important;
+    border: 1px solid var(--border) !important;
+    border-radius: 8px !important;
+    background: var(--surface) !important;
+    box-shadow: 0 8px 28px rgba(23, 32, 30, 0.055);
+}
+.panel-heading {
+    display: flex;
+    align-items: baseline;
+    justify-content: space-between;
+    min-height: 56px;
+    padding: 17px 18px 15px;
+    border-bottom: 1px solid var(--border);
+}
+.panel-heading h2 {
+    display: flex;
+    align-items: center;
+    gap: 9px;
+    margin: 0;
+    color: var(--text);
+    font-size: 0.9rem;
+    font-weight: 700;
+    line-height: 1.2;
+}
+.panel-heading span {
+    color: var(--muted);
+    font-size: 0.75rem;
+}
+.panel-heading .section-index {
+    color: var(--accent);
+    font-size: 0.7rem;
+    font-variant-numeric: tabular-nums;
+}
+.source-content,
+.viewer-content {
+    margin: 0 !important;
+    padding: 12px !important;
+    gap: 0 !important;
+    background: var(--surface) !important;
+}
+#source-image {
+    min-height: 510px;
+    overflow: hidden;
+    border: 1px solid var(--border) !important;
+    border-radius: 6px !important;
+    background: var(--surface-muted) !important;
+}
+#source-image > div,
+#source-image .wrap {
+    border-radius: 6px !important;
+    background: var(--surface-muted) !important;
+}
+#source-image p,
+#source-image span,
+#source-image button:not(.primary) {
+    color: var(--muted) !important;
+}
+.source-actions {
+    margin: 0 !important;
+    padding: 0 12px 12px;
+    border: 0 !important;
+    background: var(--surface) !important;
+}
+#reconstruct-button {
+    min-height: 46px;
+    border-color: var(--primary) !important;
+    background: var(--primary) !important;
+    color: #ffffff !important;
+    font-weight: 700;
+    box-shadow: 0 4px 12px rgba(8, 127, 98, 0.16);
+}
+#reconstruct-button:hover {
+    border-color: var(--primary-hover) !important;
+    background: var(--primary-hover) !important;
+}
+.splat-shell,
+.viewer-state,
+.splat-frame {
+    width: 100%;
+    height: min(62vh, 510px);
+    min-height: 510px;
+    background: var(--viewer);
+}
+.splat-shell {
+    overflow: hidden;
+    border-radius: 6px;
+}
+.viewer-host {
+    position: relative;
+}
+.viewer-state {
+    display: flex;
+    flex-direction: column;
+    align-items: center;
+    justify-content: center;
+    width: 100%;
+    height: 100%;
+    gap: 8px;
+    color: #f4f7f6;
+    text-align: center;
+}
+.viewer-state strong {
+    color: #f4f7f6 !important;
+    font-size: 0.9rem;
+    font-weight: 650;
+}
+.viewer-state span {
+    color: #94a19d !important;
+    font-size: 0.75rem;
+}
+.viewer-idle {
+    width: 32px;
+    height: 2px;
+    margin-bottom: 7px;
+    background: #46514e;
+}
+.viewer-loader {
+    width: 34px;
+    height: 34px;
+    margin-bottom: 7px;
+    border: 3px solid #35413d;
+    border-top-color: #34d399;
+    border-radius: 50%;
+    animation: viewer-spin 0.9s linear infinite;
+}
+.viewer-error {
+    width: 32px;
+    height: 3px;
+    margin-bottom: 7px;
+    background: #f87171;
+}
+@keyframes viewer-spin {
+    to { transform: rotate(360deg); }
+}
+@media (prefers-reduced-motion: reduce) {
+    .viewer-loader { animation-duration: 1.8s; }
+}
+.splat-frame {
+    display: block;
+    border: 0;
+}
+.viewer-host .splat-frame {
+    opacity: 0;
+    transition: opacity 180ms ease-out;
+}
+.viewer-client-state {
+    position: absolute;
+    inset: 0;
+    z-index: 2;
+    transition: opacity 180ms ease-out, visibility 180ms ease-out;
+}
+.viewer-client-error,
+.viewer-client-timeout {
+    display: none;
+}
+.viewer-host[data-viewer-state="ready"] .splat-frame {
+    opacity: 1;
+}
+.viewer-host[data-viewer-state="ready"] .viewer-client-state {
+    visibility: hidden;
+    opacity: 0;
+}
+.viewer-host[data-viewer-state="error"] .viewer-client-loading,
+.viewer-host[data-viewer-state="timeout"] .viewer-client-loading {
+    display: none;
+}
+.viewer-host[data-viewer-state="error"] .viewer-client-error,
+.viewer-host[data-viewer-state="timeout"] .viewer-client-timeout {
+    display: flex;
+}
+#viewer-output {
+    margin: 0 !important;
+    padding: 0 !important;
+    border: 0 !important;
+}
+.download-bar {
+    margin: 0 !important;
+    padding: 0 12px 12px;
+    border: 0 !important;
+    gap: 10px;
+    background: var(--surface) !important;
+}
+.artifact-button {
+    min-height: 42px;
+    border-color: var(--border-strong) !important;
+    background: var(--surface-muted) !important;
+    color: var(--text) !important;
+    font-weight: 650;
+}
+.artifact-button:hover {
+    border-color: var(--accent) !important;
+    background: #ffffff !important;
+}
+#examples-section {
+    gap: 11px !important;
+    margin: 0 0 36px !important;
+    padding: 4px 2px 0;
+}
+#example-gallery {
+    overflow: hidden;
+    padding: 0 !important;
+    border: 0 !important;
+    border-radius: 8px !important;
+    background: transparent !important;
+}
+#example-gallery .gallery-container,
+#example-gallery .grid-wrap {
+    height: 180px !important;
+    min-height: 0 !important;
+}
+#example-gallery .grid-wrap {
+    overflow: hidden !important;
+    padding: 0 !important;
+}
+#example-gallery .grid-container {
+    height: 100% !important;
+    grid-template-columns: repeat(4, minmax(0, 1fr)) !important;
+    grid-template-rows: minmax(0, 1fr) !important;
+    grid-auto-rows: minmax(0, 1fr) !important;
+    gap: 10px !important;
+}
+#example-gallery .gallery-item {
+    min-width: 0 !important;
+    overflow: hidden;
+    border-radius: 7px !important;
+}
+#example-gallery img {
+    object-fit: cover !important;
+}
+#example-gallery button {
+    color: var(--text) !important;
+}
+@media (max-width: 980px) {
+    #app-main { width: calc(100% - 28px); }
+    #workspace { flex-direction: column; }
+    #workspace > .tool-panel { width: 100% !important; }
+    .splat-shell, .viewer-state, .splat-frame {
+        height: 56vh;
+        min-height: 420px;
+    }
+    #source-image { min-height: 420px; }
+}
+@media (max-width: 560px) {
+    #app-main { width: calc(100% - 20px); }
+    #app-title { padding: 22px 2px 15px; }
+    #app-title h1 { font-size: 1.7rem; }
+    #workspace { margin: 14px 0 22px !important; gap: 12px; }
+    .panel-heading { min-height: 52px; padding: 14px; }
+    .source-content, .viewer-content { padding: 9px !important; }
+    .source-actions, .download-bar { padding: 0 9px 9px; }
+    .splat-shell, .viewer-state, .splat-frame { min-height: 360px; }
+    #source-image { min-height: 360px; }
+    #examples-section { margin-bottom: 24px !important; padding: 0; }
+}
+"""
+
+
+def build_viewer_iframe(viewer_html: Path) -> str:
+    """Build a same-origin iframe for one generated standalone viewer."""
+    encoded_path = quote(str(viewer_html.resolve()), safe="/")
+    source = html.escape(f"/gradio_api/file={encoded_path}", quote=True)
+    return (
+        '<div class="splat-shell viewer-host" '
+        'data-infinisplat-viewer data-viewer-state="loading">'
+        '<div class="viewer-state viewer-client-state viewer-client-loading">'
+        '<div class="viewer-loader"></div>'
+        "<strong>Initializing 3D viewer</strong>"
+        "<span>Uploading scene data to WebGL</span>"
+        "</div>"
+        '<div class="viewer-state viewer-client-state viewer-client-error">'
+        '<div class="viewer-error"></div>'
+        "<strong>Viewer initialization failed</strong>"
+        "<span>Check the browser console for details</span>"
+        "</div>"
+        '<div class="viewer-state viewer-client-state viewer-client-timeout">'
+        '<div class="viewer-loader"></div>'
+        "<strong>Viewer is still loading</strong>"
+        "<span>Large scenes can take longer on this device</span>"
+        "</div>"
+        '<iframe class="splat-frame" '
+        f'src="{source}" '
+        'allow="fullscreen; xr-spatial-tracking" '
+        'title="Interactive Gaussian scene" '
+        'loading="eager"></iframe>'
+        "</div>"
+    )
+
+
+def build_viewer_status(title: str, detail: str, indicator: str) -> str:
+    """Build one fixed-height viewer status surface."""
+    return (
+        '<div class="splat-shell">'
+        '<div class="viewer-state">'
+        f'<div class="viewer-{indicator}"></div>'
+        f"<strong>{html.escape(title)}</strong>"
+        f"<span>{html.escape(detail)}</span>"
+        "</div>"
+        "</div>"
+    )
+
+
+def show_reconstructing_viewer() -> str:
+    """Show the model inference stage before entering the GPU queue."""
+    return build_viewer_status(
+        "Reconstructing scene",
+        "Running model inference",
+        "loader",
+    )
+
+
+def show_exporting_viewer() -> str:
+    """Show the CPU export stage after inference completes."""
+    return build_viewer_status(
+        "Preparing viewer",
+        "Encoding PLY and optimized viewer",
+        "loader",
+    )
+
+
+def show_failed_viewer() -> str:
+    """Show a terminal viewer state when a queued stage fails."""
+    return build_viewer_status(
+        "Reconstruction stopped",
+        "See the error message for details",
+        "error",
+    )
+
+
+def select_example(evt: gr.SelectData) -> str:
+    """Return the source path selected from the example gallery."""
+    index = evt.index[0] if isinstance(evt.index, tuple) else evt.index
+    return RGB_EXAMPLES[int(index)][0]
+
+
+def configure_runtime(runtime: InfiniSplatRuntime) -> None:
+    """Register the process-wide read-only inference runtime before serving."""
+    global _runtime
+    if _runtime is not None and _runtime is not runtime:
+        raise RuntimeError("The InfiniSplat runtime is already configured.")
+    _runtime = runtime
+
+
+@spaces.GPU(duration=30)
+def reconstruct(image_path: str | None) -> str:
+    """Run one GPU reconstruction and return a CPU artifact path."""
+    if image_path is None:
+        raise gr.Error("Please upload an image.")
+    if _runtime is None:
+        raise RuntimeError("The InfiniSplat runtime is not configured.")
+    request_dir = OUTPUT_ROOT / uuid.uuid4().hex
+    artifact_path = _runtime.infer_to_artifact(
+        image_path=Path(image_path),
+        artifact_path=request_dir / "gaussians.pt",
+    )
+    return str(artifact_path)
+
+
+def export_results(artifact_path: str) -> tuple[str, str, str]:
+    """Export one CPU artifact for browser viewing and download."""
+    internal_artifact = Path(artifact_path)
+    exported = export_gaussian_artifact(
+        artifact_path=internal_artifact,
+        output_dir=internal_artifact.parent,
+    )
+    internal_artifact.unlink()
+    return (
+        build_viewer_iframe(exported.viewer_html),
+        str(exported.scene_ply),
+        str(exported.standalone_html),
+    )
+
+
+def create_demo(runtime: InfiniSplatRuntime) -> gr.Blocks:
+    """Create the public RGB reconstruction interface."""
+    configure_runtime(runtime)
+    empty_viewer = build_viewer_status(
+        "Ready",
+        "No reconstruction yet",
+        "idle",
+    )
+
+    with gr.Blocks(
+        title="InfiniSplat",
+        delete_cache=(3600, 3600),
+        analytics_enabled=False,
+    ) as demo:
+        artifact_state = gr.State()
+        with gr.Column(elem_id="app-main"):
+            gr.HTML("<h1>InfiniSplat</h1>", elem_id="app-title")
+
+            with gr.Row(equal_height=True, elem_id="workspace"):
+                with gr.Column(
+                    scale=2,
+                    min_width=300,
+                    elem_classes=["tool-panel", "source-panel"],
+                ):
+                    gr.HTML(
+                        '<div class="panel-heading">'
+                        '<h2><span class="section-index">01</span>Input image</h2>'
+                        "<span>RGB</span>"
+                        "</div>"
+                    )
+                    with gr.Column(elem_classes="source-content"):
+                        image_input = gr.Image(
+                            label="Source image",
+                            show_label=False,
+                            type="filepath",
+                            sources=["upload"],
+                            buttons=["fullscreen"],
+                            height=510,
+                            elem_id="source-image",
+                        )
+                    with gr.Row(elem_classes="source-actions"):
+                        reconstruct_button = gr.Button(
+                            "Reconstruct scene",
+                            variant="primary",
+                            elem_id="reconstruct-button",
+                        )
+                with gr.Column(
+                    scale=3,
+                    min_width=360,
+                    elem_classes=["tool-panel", "viewer-panel"],
+                ):
+                    gr.HTML(
+                        '<div class="panel-heading">'
+                        '<h2><span class="section-index">02</span>Scene viewer</h2>'
+                        "<span>Interactive</span>"
+                        "</div>"
+                    )
+                    with gr.Column(elem_classes="viewer-content"):
+                        viewer = gr.HTML(
+                            empty_viewer,
+                            label="Interactive Gaussian scene",
+                            elem_id="viewer-output",
+                        )
+                    with gr.Row(elem_classes="download-bar"):
+                        ply_download = gr.DownloadButton(
+                            "Download PLY",
+                            size="sm",
+                            elem_classes="artifact-button",
+                        )
+                        html_download = gr.DownloadButton(
+                            "Download HTML viewer",
+                            size="sm",
+                            elem_classes="artifact-button",
+                        )
+
+            with gr.Column(elem_id="examples-section"):
+                gr.HTML(
+                    '<div class="section-heading">'
+                    "<h2>Examples</h2><span>4 scenes</span>"
+                    "</div>"
+                )
+                example_gallery = gr.Gallery(
+                    value=RGB_EXAMPLES,
+                    label="Examples",
+                    show_label=False,
+                    container=False,
+                    columns=4,
+                    rows=1,
+                    height=180,
+                    allow_preview=False,
+                    object_fit="cover",
+                    buttons=[],
+                    interactive=False,
+                    elem_id="example-gallery",
+                )
+
+        example_gallery.select(
+            fn=select_example,
+            inputs=None,
+            outputs=[image_input],
+            queue=False,
+            show_progress="hidden",
+            api_name=False,
+        )
+        loading_event = reconstruct_button.click(
+            fn=show_reconstructing_viewer,
+            inputs=None,
+            outputs=[viewer],
+            queue=False,
+            show_progress="hidden",
+            api_name=False,
+        )
+        reconstruction_event = loading_event.then(
+            fn=reconstruct,
+            inputs=[image_input],
+            outputs=[artifact_state],
+            concurrency_limit=1,
+            concurrency_id="infinisplat-gpu",
+            show_progress="hidden",
+            api_name=False,
+        )
+        exporting_event = reconstruction_event.success(
+            fn=show_exporting_viewer,
+            inputs=None,
+            outputs=[viewer],
+            queue=False,
+            show_progress="hidden",
+            api_name=False,
+        )
+        export_event = exporting_event.success(
+            fn=export_results,
+            inputs=[artifact_state],
+            outputs=[viewer, ply_download, html_download],
+            show_progress="hidden",
+            api_name=False,
+        )
+        reconstruction_event.failure(
+            fn=show_failed_viewer,
+            inputs=None,
+            outputs=[viewer],
+            queue=False,
+            show_progress="hidden",
+            api_name=False,
+        )
+        export_event.failure(
+            fn=show_failed_viewer,
+            inputs=None,
+            outputs=[viewer],
+            queue=False,
+            show_progress="hidden",
+            api_name=False,
+        )
+
+    demo.queue(max_size=8, default_concurrency_limit=1)
+    return demo

src/demo/infer_batch_images.py

@@ -0,0 +1,486 @@
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import shutil
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import torch
+from rich.progress import BarColumn, Progress, SpinnerColumn, TaskProgressColumn, TextColumn
+
+from src.demo.infer_single_image import (
+    _resolve_checkpoint_path,
+    _resolve_device,
+    filter_final_gaussian_floaters,
+    load_demo_config,
+    load_demo_image_bundle,
+    load_demo_model,
+    load_prompt_depth_tensors,
+    patch_supersplat_html_auto_rotate,
+    render_novel_view_video_from_single_view,
+    run_single_image_inference,
+    scale_intrinsics_px,
+    validate_prompt_configuration,
+)
+from src.model.decoder.decoder_gsplat import is_gsplat_available
+from src.utils.gaussians import save_ply
+
+MODE_EXPERIMENTS = {
+    "rgb": "infinisplat_hypersim_rgb",
+    "lidar": "infinisplat_hypersim_lidar",
+}
+MODE_CHECKPOINTS = {
+    "rgb": Path("checkpoints/infinisplat_rgb.ckpt"),
+    "lidar": Path("checkpoints/infinisplat_lidar.ckpt"),
+}
+MODE_INPUT_DIRS = {
+    "rgb": Path("examples/data/rgb_demo"),
+    "lidar": Path("examples/data/lidar_demo"),
+}
+DEFAULT_OUTPUT_ROOT = Path("outputs/demo")
+DEFAULT_MAX_RENDER_LONG_EDGE = 3840
+DEFAULT_MAX_RENDER_PIXELS = 3840 * 2160
+SPLAT_TRANSFORM = os.environ.get("SPLAT_TRANSFORM", "splat-transform")
+VIEWER_SETTINGS = Path(__file__).resolve().parents[2] / "config" / "viewer_settings.json"
+IMAGE_EXTENSIONS = (".jpg", ".jpeg", ".png", ".bmp", ".webp")
+PROMPT_DEPTH_EXTENSIONS = (".npz", ".npy", ".png", ".h5", ".hdf5", ".exr")
+
+
+@dataclass(frozen=True)
+class CasePaths:
+    """Output paths for one image case.
+
+    Args:
+        case_dir: Directory containing all artifacts for this input image.
+        scene_ply: Final Gaussian PLY path.
+        video: Novel-view video path.
+        html: SuperSplat HTML path.
+    """
+
+    case_dir: Path
+    scene_ply: Path
+    video: Path
+    html: Path
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Run InfiniSplat inference on one image or a directory of images "
+            "without reloading model weights for every image."
+        )
+    )
+    parser.add_argument(
+        "--mode",
+        choices=tuple(MODE_EXPERIMENTS),
+        default="rgb",
+        help="Inference mode. Defaults to rgb.",
+    )
+    parser.add_argument(
+        "--checkpoint",
+        type=Path,
+        default=None,
+        help="Checkpoint override. Defaults to the released checkpoint for --mode.",
+    )
+    parser.add_argument(
+        "--input",
+        dest="input_path",
+        type=Path,
+        default=None,
+        metavar="PATH",
+        help="Input image or directory. Defaults to the bundled examples for --mode.",
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=None,
+        help="Root output directory. Defaults to outputs/demo/<mode>.",
+    )
+    parser.add_argument("--recursive", action="store_true", help="Search an input directory recursively.")
+    parser.add_argument("--limit", type=int, default=0, help="Maximum number of selected images to process; 0 means no cap.")
+    parser.add_argument("--overwrite", action="store_true", help="Recompute outputs that already exist.")
+
+    parser.add_argument("--device", type=str, default="auto")
+    camera_group = parser.add_mutually_exclusive_group()
+    camera_group.add_argument("--intrinsics-file", type=Path, default=None)
+    camera_group.add_argument("--focal-px", type=float, default=None)
+    camera_group.add_argument("--focal-mm", type=float, default=None)
+
+    prompt_group = parser.add_mutually_exclusive_group()
+    prompt_group.add_argument("--prompt-depth", type=Path, default=None)
+    prompt_group.add_argument(
+        "--prompt-depth-dir",
+        type=Path,
+        default=None,
+        help="Directory containing per-image prompt depth files with matching stems.",
+    )
+    parser.add_argument(
+        "--disable-floater-filter",
+        action="store_true",
+        help="Disable final Gaussian floater filtering.",
+    )
+
+    parser.add_argument("--no-video", action="store_true", help="Skip novel-view video rendering.")
+    parser.add_argument(
+        "--no-export-html",
+        dest="export_html",
+        action="store_false",
+        default=True,
+        help="Disable SuperSplat HTML export.",
+    )
+    return parser.parse_args()
+
+
+def _resolve_output_dir(args: argparse.Namespace) -> Path:
+    """Resolve the root output directory for this batch run."""
+    if args.output_dir is not None:
+        return args.output_dir
+    return DEFAULT_OUTPUT_ROOT / args.mode
+
+
+def _collect_images(args: argparse.Namespace) -> list[Path]:
+    """Collect input image paths in deterministic order."""
+    input_path = args.input_path or MODE_INPUT_DIRS[args.mode]
+    if input_path.is_file():
+        if input_path.suffix.lower() not in IMAGE_EXTENSIONS:
+            raise ValueError(f"Unsupported input image extension: {input_path}")
+        images = [input_path]
+    elif input_path.is_dir():
+        iterator = input_path.rglob("*") if args.recursive else input_path.iterdir()
+        images = [path for path in iterator if path.is_file() and path.suffix.lower() in IMAGE_EXTENSIONS]
+    else:
+        raise FileNotFoundError(f"Input image or directory not found: {input_path}")
+
+    images = sorted(images)
+    if not images:
+        raise FileNotFoundError("No input images were found.")
+
+    if args.limit < 0:
+        raise ValueError("--limit must be non-negative.")
+    if args.limit > 0:
+        images = images[: args.limit]
+    return images
+
+
+def _resolve_prompt_depth_path(args: argparse.Namespace, image_path: Path) -> Path | None:
+    """Resolve the prompt depth path for one image.
+
+    Args:
+        args: Parsed batch arguments.
+        image_path: RGB image path.
+
+    Returns:
+        Prompt depth path for this image, or None when prompt depth is disabled.
+    """
+    if args.prompt_depth is not None:
+        return args.prompt_depth
+    prompt_depth_dir = args.prompt_depth_dir
+    if prompt_depth_dir is None and args.mode == "lidar":
+        input_path = args.input_path or MODE_INPUT_DIRS[args.mode]
+        prompt_depth_dir = input_path if input_path.is_dir() else input_path.parent
+    if prompt_depth_dir is None:
+        return None
+
+    candidates = [
+        prompt_depth_dir / f"{image_path.stem}{ext}"
+        for ext in PROMPT_DEPTH_EXTENSIONS
+    ]
+    for candidate in candidates:
+        if candidate.exists() and candidate.resolve() != image_path.resolve():
+            return candidate
+    raise FileNotFoundError(
+        "Prompt depth file not found for "
+        f"{image_path}. Tried: {', '.join(str(candidate) for candidate in candidates)}"
+    )
+
+
+def _case_paths(output_dir: Path, image_path: Path) -> CasePaths:
+    """Create deterministic artifact paths for one image."""
+    case_dir = output_dir / image_path.stem
+    stem = image_path.stem
+    return CasePaths(
+        case_dir=case_dir,
+        scene_ply=case_dir / f"{stem}.ply",
+        video=case_dir / f"{stem}.mp4",
+        html=case_dir / f"{stem}.html",
+    )
+
+
+def _resolve_video_render_geometry(
+    intrinsics_px: torch.Tensor,
+    original_width: int,
+    original_height: int,
+) -> tuple[tuple[int, int], torch.Tensor]:
+    """Resolve capped video render shape and scaled camera intrinsics.
+
+    Args:
+        intrinsics_px: Pixel-space camera intrinsics with shape [3, 3].
+        original_width: Original image width in pixels.
+        original_height: Original image height in pixels.
+    Returns:
+        A tuple containing render image shape as (height, width) and pixel-space
+        intrinsics with shape [3, 3].
+    """
+    if original_width <= 0 or original_height <= 0:
+        raise ValueError(f"Invalid original image size: {original_width}x{original_height}")
+
+    scale = min(
+        1.0,
+        float(DEFAULT_MAX_RENDER_LONG_EDGE) / float(max(original_width, original_height)),
+        math.sqrt(float(DEFAULT_MAX_RENDER_PIXELS) / float(original_width * original_height)),
+    )
+
+    if scale >= 1.0:
+        return (original_height, original_width), intrinsics_px
+
+    render_width = max(2, int(math.floor(original_width * scale)))
+    render_height = max(2, int(math.floor(original_height * scale)))
+    # Keep video dimensions even for common yuv420 encoders.
+    render_width = max(2, render_width - (render_width % 2))
+    render_height = max(2, render_height - (render_height % 2))
+    render_intrinsics_px = scale_intrinsics_px(
+        intrinsics_px=intrinsics_px,
+        src_width=original_width,
+        src_height=original_height,
+        dst_width=render_width,
+        dst_height=render_height,
+    )
+    return (render_height, render_width), render_intrinsics_px
+
+
+def _expected_artifacts_done(paths: CasePaths, args: argparse.Namespace) -> bool:
+    """Return whether all requested artifacts already exist."""
+    expected = [paths.scene_ply]
+    if not args.no_video:
+        expected.append(paths.video)
+    if args.export_html:
+        expected.append(paths.html)
+    return all(path.exists() for path in expected)
+
+
+def _needs_only_conversion(paths: CasePaths, args: argparse.Namespace) -> bool:
+    """Return whether inference is done but requested converted outputs are missing."""
+    if not paths.scene_ply.exists():
+        return False
+    if not args.no_video and not paths.video.exists():
+        return False
+    return not _expected_artifacts_done(paths, args)
+
+
+def _disable_unavailable_optional_outputs(args: argparse.Namespace) -> None:
+    """Skip optional outputs whose external dependencies are unavailable."""
+    if not args.no_video and not is_gsplat_available():
+        print("[batch] Skipping video rendering; optional gsplat is unavailable.")
+        args.no_video = True
+    if (
+        args.export_html
+        and shutil.which(SPLAT_TRANSFORM) is None
+        and not Path(SPLAT_TRANSFORM).exists()
+    ):
+        print("[batch] Skipping HTML export; optional converter is unavailable.")
+        args.export_html = False
+
+
+def _build_splat_transform_command(
+    scene_ply: Path,
+    output_path: Path,
+    viewer_settings: Path,
+) -> list[str]:
+    """Build the fixed SH0 HTML conversion command."""
+    command = [
+        SPLAT_TRANSFORM,
+        "-w",
+        "--viewer-settings",
+        str(viewer_settings),
+        str(scene_ply),
+        "--filter-harmonics",
+        "0",
+        str(output_path),
+    ]
+    return command
+
+
+def _run_splat_transform(
+    scene_ply: Path,
+    output_path: Path,
+    viewer_settings: Path,
+) -> None:
+    """Convert one Gaussian PLY into a quiet, paused HTML viewer."""
+    command = _build_splat_transform_command(scene_ply, output_path, viewer_settings)
+    subprocess.run(command, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+    patch_supersplat_html_auto_rotate(output_path)
+
+
+def _convert_scene_if_requested(
+    export_html: bool,
+    paths: CasePaths,
+    viewer_settings: Path | None,
+) -> None:
+    """Convert scene.ply into the default HTML viewer when requested."""
+    if not export_html:
+        return
+    if viewer_settings is None:
+        raise RuntimeError("Viewer settings are required for HTML export.")
+    _run_splat_transform(paths.scene_ply, paths.html, viewer_settings)
+
+
+def _validate_batch_prompt_configuration(cfg: Any, args: argparse.Namespace) -> bool:
+    """Validate prompt configuration for batch inference."""
+    prompt_depth_enabled = (
+        args.mode == "lidar"
+        or args.prompt_depth is not None
+        or args.prompt_depth_dir is not None
+    )
+    return validate_prompt_configuration(cfg, prompt_depth_enabled)
+
+
+@torch.inference_mode()
+def _run_one_image(
+    args: argparse.Namespace,
+    image_path: Path,
+    paths: CasePaths,
+    encoder: Any,
+    decoder: Any,
+    prompt_enabled: bool,
+    device: torch.device,
+    viewer_settings: Path | None,
+) -> None:
+    """Run inference and export artifacts for one input image."""
+    paths.case_dir.mkdir(parents=True, exist_ok=True)
+
+    image_bundle = load_demo_image_bundle(
+        image_path=image_path,
+        focal_length_px=args.focal_px,
+        focal_length_mm=args.focal_mm,
+        intrinsics_override_path=args.intrinsics_file,
+    )
+    original_height, original_width = image_bundle.original_image_shape
+    _, inference_height, inference_width = image_bundle.inference_image.shape
+    render_image_shape, render_intrinsics_px = _resolve_video_render_geometry(
+        intrinsics_px=image_bundle.original_intrinsics.intrinsics_px,
+        original_width=original_width,
+        original_height=original_height,
+    )
+    prompt_inputs = None
+    if prompt_enabled:
+        prompt_depth_path = _resolve_prompt_depth_path(args, image_path)
+        if prompt_depth_path is None:
+            raise ValueError("Prompt-conditioned inference requires --prompt-depth or --prompt-depth-dir.")
+        prompt_inputs = load_prompt_depth_tensors(
+            prompt_depth_path=prompt_depth_path,
+            image_shape=(inference_height, inference_width),
+        )
+
+    encoder_output = run_single_image_inference(
+        encoder=encoder,
+        image=image_bundle.inference_image,
+        intrinsics_px=image_bundle.inference_intrinsics.intrinsics_px,
+        device=device,
+        prompt_inputs=prompt_inputs,
+    )
+
+    final_gaussians = encoder_output["gaussians"]
+    if not args.disable_floater_filter:
+        final_gaussians = filter_final_gaussian_floaters(final_gaussians)
+
+    save_ply(
+        gaussians=final_gaussians,
+        f_px=image_bundle.inference_intrinsics.focal_length_px,
+        image_shape=(inference_height, inference_width),
+        path=paths.scene_ply,
+    )
+
+    if not args.no_video:
+        render_novel_view_video_from_single_view(
+            decoder=decoder,
+            gaussians=final_gaussians,
+            render_intrinsics_px=render_intrinsics_px,
+            render_image_shape=render_image_shape,
+            output_path=paths.video,
+        )
+
+    _convert_scene_if_requested(args.export_html, paths, viewer_settings)
+
+
+def run_batch(args: argparse.Namespace) -> dict[str, Any]:
+    """Run the batch inference pipeline."""
+    images = _collect_images(args)
+    output_dir = _resolve_output_dir(args)
+    output_dir.mkdir(parents=True, exist_ok=True)
+    _disable_unavailable_optional_outputs(args)
+    viewer_settings = VIEWER_SETTINGS if args.export_html else None
+
+    succeeded = 0
+    skipped = 0
+    progress = Progress(
+        SpinnerColumn(style="cyan"),
+        TextColumn("[bold cyan]{task.description}"),
+        BarColumn(bar_width=36, complete_style="cyan", finished_style="green"),
+        TaskProgressColumn(),
+        TextColumn("{task.completed:.0f}/{task.total:.0f}"),
+    )
+    with progress:
+        task = progress.add_task("Preparing model", total=len(images))
+        cfg = load_demo_config(MODE_EXPERIMENTS[args.mode])
+        prompt_enabled = _validate_batch_prompt_configuration(cfg, args)
+        checkpoint_path = _resolve_checkpoint_path(
+            args.checkpoint or MODE_CHECKPOINTS[args.mode]
+        )
+        device = _resolve_device(args.device)
+        encoder, decoder = load_demo_model(cfg=cfg, checkpoint_path=checkpoint_path, device=device)
+        progress.update(task, description="Running inference")
+
+        for image_path in images:
+            paths = _case_paths(output_dir, image_path)
+            if not args.overwrite and _expected_artifacts_done(paths, args):
+                skipped += 1
+                progress.advance(task)
+                continue
+
+            try:
+                if not args.overwrite and _needs_only_conversion(paths, args):
+                    _convert_scene_if_requested(
+                        args.export_html,
+                        paths,
+                        viewer_settings,
+                    )
+                else:
+                    _run_one_image(
+                        args=args,
+                        image_path=image_path,
+                        paths=paths,
+                        encoder=encoder,
+                        decoder=decoder,
+                        prompt_enabled=prompt_enabled,
+                        device=device,
+                        viewer_settings=viewer_settings,
+                    )
+                succeeded += 1
+            finally:
+                if device.type == "cuda":
+                    torch.cuda.empty_cache()
+                progress.advance(task)
+
+        progress.update(task, description="Complete")
+
+    return {
+        "status": "success",
+        "succeeded": succeeded,
+        "skipped": skipped,
+        "output_dir": str(output_dir),
+    }
+
+
+def main() -> None:
+    args = _parse_args()
+    result = run_batch(args)
+    print(json.dumps(result, indent=2, ensure_ascii=False))
+
+
+if __name__ == "__main__":
+    main()

src/demo/infer_single_image.py

@@ -0,0 +1,889 @@
+from __future__ import annotations
+
+import math
+import os
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import imageio.v2 as imageio
+import numpy as np
+
+os.environ["OPENCV_IO_ENABLE_OPENEXR"] = "1"
+
+import cv2
+import torch
+import torchvision.transforms as tf
+from einops import rearrange, repeat
+from hydra import compose, initialize_config_dir
+from hydra.core.global_hydra import GlobalHydra
+from omegaconf import OmegaConf
+from PIL import ExifTags, Image, ImageOps, TiffTags
+from scipy.spatial import cKDTree
+
+from src.demo.config import RootCfg, load_typed_root_config
+from src.model.decoder import Decoder, get_decoder
+from src.model.encoder import Encoder, get_encoder
+from src.utils.gaussians import Gaussians3D
+from src.utils.io import save_video
+
+DEFAULT_FOCAL_35MM_MM = 30.0
+INFERENCE_HEIGHT = 1152
+INFERENCE_WIDTH = 1536
+DEFAULT_VIDEO_FPS = 10
+DEFAULT_VIDEO_FRAMES = 60
+DEFAULT_RENDER_CHUNK_SIZE = 8
+PROMPT_DEPTH_EPS = 1e-6
+PROMPT_DISPARITY_MIN_DEPTH = 1e-2
+
+
+@dataclass(frozen=True)
+class ResolvedIntrinsics:
+    """Resolved intrinsics for one input image.
+
+    Args:
+        intrinsics_px: Pixel-space intrinsics with shape [3, 3].
+        focal_length_px: Pixel focal length used for export and trajectory sizing.
+    """
+
+    intrinsics_px: torch.Tensor
+    focal_length_px: float
+
+
+@dataclass(frozen=True)
+class PromptInputs:
+    """Prompt-conditioned depth inputs for one single-view demo sample.
+
+    Args:
+        prompt_disparity: Sparse prompt disparity with shape [1, H, W].
+        prompt_mask: Prompt validity mask with shape [1, H, W].
+    """
+
+    prompt_disparity: torch.Tensor
+    prompt_mask: torch.Tensor
+
+
+@dataclass(frozen=True)
+class DemoImageBundle:
+    """Original/render image data and inference image data for one demo sample.
+
+    Args:
+        original_image_shape: Original image shape as (height, width).
+        inference_image: Inference-resolution image tensor with shape [3, H_inf, W_inf].
+        original_intrinsics: Pixel-space intrinsics resolved in original image space.
+        inference_intrinsics: Pixel-space intrinsics resolved in inference image space.
+    """
+
+    original_image_shape: tuple[int, int]
+    inference_image: torch.Tensor
+    original_intrinsics: ResolvedIntrinsics
+    inference_intrinsics: ResolvedIntrinsics
+
+
+def _resolve_device(device_name: str) -> torch.device:
+    if device_name == "auto":
+        return torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    return torch.device(device_name)
+
+
+def _as_float(value: Any) -> float | None:
+    if value is None:
+        return None
+    if isinstance(value, tuple) and len(value) == 2:
+        numerator, denominator = value
+        if float(denominator) == 0:
+            return None
+        return float(numerator) / float(denominator)
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def convert_focallength_mm_to_px(width: float, height: float, focal_mm: float) -> float:
+    """Convert 35mm-equivalent focal length in millimeters to pixels."""
+    return focal_mm * math.sqrt(width**2.0 + height**2.0) / math.sqrt(36.0**2 + 24.0**2)
+
+
+def build_intrinsics_from_focal_px(
+    focal_length_px: float,
+    width: int,
+    height: int,
+) -> torch.Tensor:
+    """Create a pixel-space OpenCV intrinsics matrix."""
+    return torch.tensor(
+        [
+            [focal_length_px, 0.0, (width - 1) / 2.0],
+            [0.0, focal_length_px, (height - 1) / 2.0],
+            [0.0, 0.0, 1.0],
+        ],
+        dtype=torch.float32,
+    )
+
+
+def normalize_intrinsics(
+    intrinsics_px: torch.Tensor,
+    width: int,
+    height: int,
+) -> torch.Tensor:
+    """Convert pixel-space intrinsics into the normalized InfiniSplat convention."""
+    intrinsics_norm = intrinsics_px.clone()
+    intrinsics_norm[0] = intrinsics_norm[0] / float(width)
+    intrinsics_norm[1] = intrinsics_norm[1] / float(height)
+    return intrinsics_norm
+
+
+def scale_intrinsics_px(
+    intrinsics_px: torch.Tensor,
+    src_width: int,
+    src_height: int,
+    dst_width: int,
+    dst_height: int,
+) -> torch.Tensor:
+    """Scale pixel-space intrinsics from one image resolution to another."""
+    scaled = intrinsics_px.clone()
+    scale_x = float(dst_width) / float(src_width)
+    scale_y = float(dst_height) / float(src_height)
+    scaled[0] = scaled[0] * scale_x
+    scaled[1] = scaled[1] * scale_y
+    return scaled
+
+
+def _extract_exif_dict(image: Image.Image) -> dict[str, Any]:
+    exif_block = image.getexif().get_ifd(0x8769)
+    exif_dict = {ExifTags.TAGS[k]: v for k, v in exif_block.items() if k in ExifTags.TAGS}
+    tiff_tags = image.getexif()
+    tiff_dict = {TiffTags.TAGS_V2[k].name: v for k, v in tiff_tags.items() if k in TiffTags.TAGS_V2}
+    return {**exif_dict, **tiff_dict}
+
+
+def _extract_exif_focal_length_px(
+    image: Image.Image,
+    width: int,
+    height: int,
+) -> float | None:
+    exif_dict = _extract_exif_dict(image)
+    focal_35mm = _as_float(
+        exif_dict.get("FocalLengthIn35mmFilm", exif_dict.get("FocalLenIn35mmFilm"))
+    )
+    if focal_35mm is None or focal_35mm < 1.0:
+        focal_mm = _as_float(exif_dict.get("FocalLength"))
+        if focal_mm is None:
+            return None
+        if focal_mm < 10.0:
+            focal_35mm = focal_mm * 8.4
+        else:
+            focal_35mm = focal_mm
+    return convert_focallength_mm_to_px(width, height, focal_35mm)
+
+
+def _load_intrinsics_override(path: Path) -> torch.Tensor:
+    cfg = OmegaConf.load(path)
+    container = OmegaConf.to_container(cfg, resolve=True)
+    if isinstance(container, dict):
+        if "intrinsics_px" in container:
+            matrix = container["intrinsics_px"]
+        elif "camera" in container and isinstance(container["camera"], dict) and "intrinsics_px" in container["camera"]:
+            matrix = container["camera"]["intrinsics_px"]
+        else:
+            raise KeyError(
+                f"Could not find a 3x3 intrinsics matrix in {path}. Expected `intrinsics_px` or `camera.intrinsics_px`."
+            )
+    else:
+        matrix = container
+    intrinsics_px = torch.tensor(matrix, dtype=torch.float32)
+    if intrinsics_px.shape != (3, 3):
+        raise ValueError(f"Expected a 3x3 intrinsics matrix in {path}, got {tuple(intrinsics_px.shape)}.")
+    return intrinsics_px
+
+
+def resolve_intrinsics(
+    image: Image.Image,
+    focal_length_px: float | None = None,
+    focal_length_mm: float | None = None,
+    intrinsics_override_path: Path | None = None,
+    default_focal_35mm_mm: float = DEFAULT_FOCAL_35MM_MM,
+) -> ResolvedIntrinsics:
+    """Resolve pixel-space and normalized intrinsics for one image."""
+    width, height = image.size
+    if intrinsics_override_path is not None:
+        intrinsics_px = _load_intrinsics_override(intrinsics_override_path)
+        resolved_focal_px = float(intrinsics_px[0, 0].item())
+    elif focal_length_px is not None:
+        intrinsics_px = build_intrinsics_from_focal_px(focal_length_px, width, height)
+        resolved_focal_px = float(focal_length_px)
+    elif focal_length_mm is not None:
+        resolved_focal_px = convert_focallength_mm_to_px(width, height, focal_length_mm)
+        intrinsics_px = build_intrinsics_from_focal_px(resolved_focal_px, width, height)
+    else:
+        exif_focal_px = _extract_exif_focal_length_px(image, width, height)
+        if exif_focal_px is not None:
+            resolved_focal_px = exif_focal_px
+            intrinsics_px = build_intrinsics_from_focal_px(resolved_focal_px, width, height)
+        else:
+            resolved_focal_px = convert_focallength_mm_to_px(width, height, default_focal_35mm_mm)
+            intrinsics_px = build_intrinsics_from_focal_px(resolved_focal_px, width, height)
+
+    return ResolvedIntrinsics(
+        intrinsics_px=intrinsics_px,
+        focal_length_px=resolved_focal_px,
+    )
+
+
+def load_demo_image_bundle(
+    image_path: Path,
+    focal_length_px: float | None = None,
+    focal_length_mm: float | None = None,
+    intrinsics_override_path: Path | None = None,
+    default_focal_35mm_mm: float = DEFAULT_FOCAL_35MM_MM,
+) -> DemoImageBundle:
+    """Load both original-resolution and inference-resolution image/intrinsics bundles."""
+    with Image.open(image_path) as image_pil:
+        image_rgb = ImageOps.exif_transpose(image_pil).convert("RGB")
+        original_width, original_height = image_rgb.size
+        original_intrinsics = resolve_intrinsics(
+            image=image_rgb,
+            focal_length_px=focal_length_px,
+            focal_length_mm=focal_length_mm,
+            intrinsics_override_path=intrinsics_override_path,
+            default_focal_35mm_mm=default_focal_35mm_mm,
+        )
+
+        target_height = INFERENCE_HEIGHT
+        target_width = INFERENCE_WIDTH
+
+        inference_image_rgb = image_rgb
+        inference_intrinsics = original_intrinsics
+        if (original_height, original_width) != (target_height, target_width):
+            inference_image_rgb = image_rgb.resize((target_width, target_height), Image.Resampling.BILINEAR)
+            inference_intrinsics_px = scale_intrinsics_px(
+                original_intrinsics.intrinsics_px,
+                src_width=original_width,
+                src_height=original_height,
+                dst_width=target_width,
+                dst_height=target_height,
+            )
+            inference_intrinsics = ResolvedIntrinsics(
+                intrinsics_px=inference_intrinsics_px,
+                focal_length_px=float(inference_intrinsics_px[0, 0].item()),
+            )
+
+        return DemoImageBundle(
+            original_image_shape=(original_height, original_width),
+            inference_image=tf.ToTensor()(inference_image_rgb),
+            original_intrinsics=original_intrinsics,
+            inference_intrinsics=inference_intrinsics,
+        )
+
+
+def _to_single_channel_depth(depth: np.ndarray, depth_path: Path) -> np.ndarray:
+    """Convert loaded depth arrays to a single-channel float32 map."""
+    depth = np.asarray(depth)
+    while depth.ndim > 3:
+        singleton_axes = [axis for axis, size in enumerate(depth.shape) if size == 1]
+        if not singleton_axes:
+            break
+        depth = np.squeeze(depth, axis=singleton_axes[0])
+    if depth.ndim == 3 and depth.shape[0] in (1, 3) and depth.shape[2] not in (1, 3):
+        depth = np.moveaxis(depth, 0, -1)
+    if depth.ndim == 2:
+        return depth.astype(np.float32)
+    if depth.ndim == 3:
+        if depth.shape[2] == 1:
+            return depth[:, :, 0].astype(np.float32)
+        if np.issubdtype(depth.dtype, np.integer) and depth.shape[2] >= 3:
+            ch0 = depth[:, :, 0]
+            ch1 = depth[:, :, 1]
+            ch2 = depth[:, :, 2]
+            if np.array_equal(ch0, ch1) and np.array_equal(ch1, ch2):
+                return ch0.astype(np.float32)
+            return (
+                ch0.astype(np.float32) * (256.0**2)
+                + ch1.astype(np.float32) * 256.0
+                + ch2.astype(np.float32)
+            )
+        return depth[:, :, 0].astype(np.float32)
+    raise ValueError(f"Unsupported depth shape {depth.shape} for file: {depth_path}")
+
+
+def _load_depth_from_png(depth_path: Path) -> np.ndarray:
+    raw = imageio.imread(depth_path)
+    depth = _to_single_channel_depth(raw, depth_path)
+    if np.issubdtype(raw.dtype, np.integer) and float(np.nanmax(depth)) > 255.0:
+        depth = depth / 1000.0
+    return depth.astype(np.float32)
+
+
+def _decode_sparse_depth(
+    mask: np.ndarray,
+    value: np.ndarray,
+    depth_path: Path,
+) -> np.ndarray:
+    mask = np.asarray(mask).astype(bool)
+    value = np.asarray(value, dtype=np.float32)
+    depth = np.zeros(mask.shape, dtype=np.float32)
+    if value.shape == mask.shape:
+        depth[mask] = value[mask]
+        return depth
+
+    value_flat = value.reshape(-1)
+    valid_count = int(mask.sum())
+    if value_flat.size < valid_count:
+        raise ValueError(
+            f"Depth value count ({value_flat.size}) is smaller than the mask valid count "
+            f"({valid_count}): {depth_path}"
+        )
+    depth[mask] = value_flat[:valid_count]
+    return depth
+
+
+def _load_depth_from_npz(depth_path: Path) -> np.ndarray:
+    with np.load(depth_path, allow_pickle=False) as npz_data:
+        if len(npz_data.files) == 0:
+            raise ValueError(f"Empty npz depth file: {depth_path}")
+        if "mask" in npz_data.files and "value" in npz_data.files:
+            return _decode_sparse_depth(
+                npz_data["mask"],
+                npz_data["value"],
+                depth_path,
+            )
+        preferred_keys = ("depth", "data", "depth_map", "arr_0")
+        key = next((k for k in preferred_keys if k in npz_data.files), npz_data.files[0])
+        depth = np.asarray(npz_data[key])
+    return _to_single_channel_depth(depth, depth_path)
+
+
+def _find_h5_dataset(node: Any) -> Any:
+    preferred_keys = ("dataset", "depth", "data")
+    for key in preferred_keys:
+        if key in node:
+            child = node[key]
+            if child.__class__.__name__ == "Dataset":
+                return child
+    for key in node.keys():
+        child = node[key]
+        if child.__class__.__name__ == "Dataset":
+            return child
+    for key in node.keys():
+        child = node[key]
+        if child.__class__.__name__ == "Group":
+            found = _find_h5_dataset(child)
+            if found is not None:
+                return found
+    return None
+
+
+def _load_depth_from_h5(depth_path: Path) -> np.ndarray:
+    import h5py
+
+    with h5py.File(depth_path, "r") as h5_file:
+        dataset = _find_h5_dataset(h5_file)
+        if dataset is None:
+            raise ValueError(f"No dataset found in h5 file: {depth_path}")
+        depth = np.asarray(dataset)
+    return _to_single_channel_depth(depth, depth_path)
+
+
+def _load_depth_from_npy(depth_path: Path) -> np.ndarray:
+    return _to_single_channel_depth(np.load(depth_path, allow_pickle=False), depth_path)
+
+
+def _load_depth_from_exr(depth_path: Path) -> np.ndarray:
+    depth = cv2.imread(str(depth_path), cv2.IMREAD_ANYCOLOR | cv2.IMREAD_ANYDEPTH)
+    if depth is None:
+        raise ValueError(f"Failed to read EXR depth file: {depth_path}")
+    return _to_single_channel_depth(depth, depth_path)
+
+
+def _read_depth_array(depth_path: Path) -> np.ndarray:
+    """Load a raw depth array from a supported prompt depth file."""
+    ext = depth_path.suffix.lower()
+    if ext == ".png":
+        return _load_depth_from_png(depth_path)
+    if ext == ".npz":
+        return _load_depth_from_npz(depth_path)
+    if ext in (".hdf5", ".h5"):
+        return _load_depth_from_h5(depth_path)
+    if ext == ".npy":
+        return _load_depth_from_npy(depth_path)
+    if ext == ".exr":
+        return _load_depth_from_exr(depth_path)
+    raise ValueError(f"Unsupported prompt depth extension `{ext}`: {depth_path}")
+
+
+def load_depth(
+    depth_path: Path,
+    tar_size: tuple[int, int],
+    num_samples: int = 1500,
+    min_prompt: int = 1,
+    max_prompt: int = 100,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+    """Load dense depth and build a sparse prompt map for InfiniDepth."""
+    depth = _read_depth_array(depth_path).astype(np.float32)
+    depth[~np.isfinite(depth)] = 0.0
+    depth = cv2.resize(depth, tar_size[::-1], interpolation=cv2.INTER_NEAREST)
+
+    depth_mask = ((depth > min_prompt) & (depth < max_prompt)).astype(np.float32)
+    valid_depth = depth * depth_mask
+    if (valid_depth > 0.1).sum() > num_samples:
+        height, width = depth.shape
+        sample_depth = valid_depth.reshape(-1)
+        nonzero_index = np.flatnonzero(sample_depth > 0.1)
+        index = np.random.permutation(nonzero_index)[:num_samples]
+        sample_mask = np.ones_like(sample_depth)
+        sample_mask[index] = 0.0
+        sample_depth[sample_mask.astype(bool)] = 0.0
+        sample_depth = sample_depth.reshape(height, width)
+    else:
+        sample_depth = valid_depth
+
+    depth_tensor = torch.from_numpy(depth).unsqueeze(0).unsqueeze(0).float()
+    sample_depth_tensor = torch.from_numpy(sample_depth).unsqueeze(0).unsqueeze(0).float()
+    depth_mask_tensor = torch.from_numpy(depth_mask).unsqueeze(0).unsqueeze(0)
+    return depth_tensor, sample_depth_tensor, depth_mask_tensor
+
+
+def load_prompt_depth_tensors(
+    prompt_depth_path: Path,
+    image_shape: tuple[int, int],
+) -> PromptInputs:
+    """Load one prompt depth file and build prompt tensors for InfiniDepth demo inference."""
+    if not prompt_depth_path.exists():
+        raise FileNotFoundError(f"Prompt depth file not found: {prompt_depth_path}")
+
+    _, prompt_depth_tensor, _ = load_depth(
+        depth_path=prompt_depth_path,
+        tar_size=image_shape,
+    )
+
+    prompt_depth = prompt_depth_tensor.squeeze(0).squeeze(0).numpy()
+    prompt_mask = (prompt_depth > PROMPT_DISPARITY_MIN_DEPTH).astype(np.float32)
+    valid_mask = prompt_mask > 0.0
+    if not valid_mask.any():
+        raise ValueError(
+            f"Prompt depth file `{prompt_depth_path}` does not contain any valid prompt pixels after resizing."
+        )
+
+    prompt_disparity = np.zeros_like(prompt_depth, dtype=np.float32)
+    prompt_disparity[valid_mask] = 1.0 / np.clip(
+        prompt_depth[valid_mask],
+        a_min=PROMPT_DEPTH_EPS,
+        a_max=None,
+    )
+
+    return PromptInputs(
+        prompt_disparity=torch.from_numpy(prompt_disparity).unsqueeze(0),
+        prompt_mask=torch.from_numpy(prompt_mask).unsqueeze(0),
+    )
+
+
+def _requires_prompt_depth(cfg: RootCfg) -> bool:
+    """Return whether the current encoder requires an explicit prompt depth input."""
+    return cfg.model.encoder.name == "infinisplat_infinidepth"
+
+
+def validate_prompt_configuration(cfg: RootCfg, prompt_depth_enabled: bool) -> bool:
+    """Validate whether prompt inputs match the selected encoder configuration."""
+    requires_prompt = _requires_prompt_depth(cfg)
+    if requires_prompt and not prompt_depth_enabled:
+        raise ValueError(
+            "`infinisplat_infinidepth` demo inference requires `--prompt-depth` because the encoder uses prompt-conditioned InfiniDepth."
+        )
+    if not requires_prompt and prompt_depth_enabled:
+        raise ValueError(
+            "`--prompt-depth` is only supported when the resolved encoder is `infinisplat_infinidepth`."
+        )
+    return requires_prompt
+
+
+def build_single_view_batch(
+    image: torch.Tensor,
+    intrinsics_px: torch.Tensor,
+    device: torch.device | str = "cpu",
+    prompt_inputs: PromptInputs | None = None,
+) -> dict[str, torch.Tensor]:
+    """Build the minimal single-view batch required by the encoder."""
+    device = torch.device(device)
+    _, height, width = image.shape
+    intrinsics_norm = normalize_intrinsics(intrinsics_px, width, height)
+    eye = torch.eye(4, dtype=torch.float32, device=device)
+    context = {
+        "image": rearrange(image.to(device=device, dtype=torch.float32), "c h w -> 1 1 c h w"),
+        "intrinsics": rearrange(intrinsics_norm.to(device=device, dtype=torch.float32), "i j -> 1 1 i j"),
+        "extrinsics": rearrange(eye, "i j -> 1 1 i j"),
+    }
+    if prompt_inputs is not None:
+        context["prompt_disparity"] = rearrange(
+            prompt_inputs.prompt_disparity.to(device=device, dtype=torch.float32),
+            "c h w -> 1 1 c h w",
+        )
+        context["prompt_mask"] = rearrange(
+            prompt_inputs.prompt_mask.to(device=device, dtype=torch.float32),
+            "c h w -> 1 1 c h w",
+        )
+    return context
+
+
+def load_demo_config(
+    experiment_name: str,
+) -> RootCfg:
+    """Compose the Hydra config used for demo inference."""
+    config_dir = Path(__file__).resolve().parents[2] / "config"
+    if GlobalHydra.instance().is_initialized():
+        GlobalHydra.instance().clear()
+    with initialize_config_dir(version_base=None, config_dir=str(config_dir)):
+        cfg_dict = compose(
+            config_name="inference",
+            overrides=[
+                f"+experiment={experiment_name}",
+            ],
+        )
+    return load_typed_root_config(cfg_dict)
+
+
+def _extract_state_dict(checkpoint_path: Path) -> dict[str, torch.Tensor]:
+    checkpoint = torch.load(checkpoint_path, map_location="cpu", weights_only=True)
+    if "state_dict" in checkpoint:
+        return checkpoint["state_dict"]
+    if isinstance(checkpoint, dict):
+        return checkpoint
+    raise TypeError(f"Unexpected checkpoint format at {checkpoint_path}.")
+
+
+def _load_prefixed_state_dict(
+    module: torch.nn.Module,
+    state_dict: dict[str, torch.Tensor],
+    prefix: str,
+) -> None:
+    module_state = {
+        key[len(prefix):]: value
+        for key, value in state_dict.items()
+        if key.startswith(prefix)
+    }
+    if not module_state:
+        if len(module.state_dict()) == 0:
+            return
+        raise KeyError(f"Did not find any weights with prefix `{prefix}` in the checkpoint.")
+    missing_keys, unexpected_keys = module.load_state_dict(module_state, strict=True)
+    if missing_keys or unexpected_keys:
+        raise RuntimeError(
+            f"Unexpected state-dict mismatch for prefix `{prefix}`: "
+            f"missing={missing_keys}, unexpected={unexpected_keys}"
+        )
+
+
+def load_demo_model(
+    cfg: RootCfg,
+    checkpoint_path: Path,
+    device: torch.device,
+) -> tuple[Encoder, Decoder]:
+    """Instantiate encoder/decoder and load checkpoint weights for demo inference."""
+    encoder = get_encoder(cfg.model.encoder)
+    decoder = get_decoder(cfg.model.decoder)
+
+    state_dict = _extract_state_dict(checkpoint_path)
+    _load_prefixed_state_dict(encoder, state_dict, "encoder.")
+    _load_prefixed_state_dict(decoder, state_dict, "decoder.")
+
+    encoder = encoder.to(device)
+    decoder = decoder.to(device)
+    encoder.eval()
+    decoder.eval()
+    return encoder, decoder
+
+
+def load_demo_encoder(
+    cfg: RootCfg,
+    checkpoint_path: Path,
+    device: torch.device,
+) -> Encoder:
+    """Instantiate the encoder and restore only its checkpoint weights."""
+    encoder = get_encoder(cfg.model.encoder)
+    state_dict = _extract_state_dict(checkpoint_path)
+    _load_prefixed_state_dict(encoder, state_dict, "encoder.")
+    encoder = encoder.to(device)
+    encoder.eval()
+    return encoder
+
+
+@torch.inference_mode()
+def run_single_image_inference(
+    encoder: Encoder,
+    image: torch.Tensor,
+    intrinsics_px: torch.Tensor,
+    device: torch.device,
+    prompt_inputs: PromptInputs | None = None,
+) -> dict[str, torch.Tensor]:
+    """Run encoder-only inference for one image."""
+    context = build_single_view_batch(
+        image=image,
+        intrinsics_px=intrinsics_px,
+        device=device,
+        prompt_inputs=prompt_inputs,
+    )
+    return encoder(context)
+
+
+def _filter_gaussians_by_mask(
+    gaussians: Gaussians3D,
+    keep_mask: torch.Tensor,
+) -> Gaussians3D:
+    """Select a single-image subset of Gaussians.
+
+    Args:
+        gaussians: Gaussian container with shape [1, N, ...].
+        keep_mask: Boolean keep mask with shape [N].
+
+    Returns:
+        Filtered Gaussian container with shape [1, M, ...].
+    """
+    return Gaussians3D(
+        mean_vectors=gaussians.mean_vectors[:, keep_mask],
+        singular_values=gaussians.singular_values[:, keep_mask],
+        quaternions=gaussians.quaternions[:, keep_mask],
+        colors=gaussians.colors[:, keep_mask],
+        opacities=gaussians.opacities[:, keep_mask],
+        covariances=gaussians.covariances[:, keep_mask] if gaussians.covariances is not None else None,
+    )
+
+
+@torch.inference_mode()
+def filter_final_gaussian_floaters(gaussians: Gaussians3D) -> Gaussians3D:
+    """Remove spatial outliers from final single-image Gaussians."""
+    if gaussians.mean_vectors.shape[0] != 1:
+        raise ValueError("Single-image floater filtering expects batch size 1.")
+
+    total = int(gaussians.mean_vectors.shape[1])
+    points = gaussians.mean_vectors[0]
+    candidate_indices = torch.nonzero(
+        torch.isfinite(points).all(dim=-1),
+        as_tuple=False,
+    ).flatten()
+    if candidate_indices.numel() == 0:
+        return gaussians
+
+    candidate_points = points[candidate_indices].detach().float().cpu().numpy()
+    if len(candidate_points) < 2:
+        return gaussians
+    neighbor_count = min(16, len(candidate_points) - 1)
+    distances, _ = cKDTree(candidate_points).query(
+        candidate_points,
+        k=neighbor_count + 1,
+    )
+    mean_distances = distances[:, 1:].mean(axis=1)
+    threshold = mean_distances.mean() + 2.5 * mean_distances.std()
+    inlier_indices = np.flatnonzero(mean_distances <= threshold)
+
+    keep_mask = torch.zeros(total, device=points.device, dtype=torch.bool)
+    if inlier_indices.size > 0:
+        inlier_indices_tensor = torch.as_tensor(
+            inlier_indices,
+            device=points.device,
+            dtype=torch.long,
+        )
+        keep_mask[candidate_indices[inlier_indices_tensor]] = True
+    if not keep_mask.any():
+        return gaussians
+    return _filter_gaussians_by_mask(gaussians, keep_mask)
+
+
+def _normalize_vector(vectors: torch.Tensor, eps: float = 1e-8) -> torch.Tensor:
+    return vectors / vectors.norm(dim=-1, keepdim=True).clamp_min(eps)
+
+
+def _build_look_at_c2w(
+    eye_positions: torch.Tensor,
+    look_at: torch.Tensor,
+    world_up: torch.Tensor,
+) -> torch.Tensor:
+    """Construct OpenCV-style camera-to-world matrices from eye positions."""
+    forward = _normalize_vector(look_at.unsqueeze(0) - eye_positions)
+    right = torch.cross(forward, world_up.unsqueeze(0).expand_as(forward), dim=-1)
+    near_parallel = right.norm(dim=-1) < 1e-6
+    if near_parallel.any():
+        fallback_up = torch.tensor([0.0, 0.0, -1.0], device=eye_positions.device, dtype=eye_positions.dtype)
+        right = torch.where(
+            near_parallel[:, None],
+            torch.cross(forward, fallback_up.unsqueeze(0).expand_as(forward), dim=-1),
+            right,
+        )
+    right = _normalize_vector(right)
+    down = _normalize_vector(torch.cross(forward, right, dim=-1))
+
+    c2w = repeat(
+        torch.eye(4, dtype=eye_positions.dtype, device=eye_positions.device),
+        "i j -> t i j",
+        t=eye_positions.shape[0],
+    ).clone()
+    c2w[:, :3, 0] = right
+    c2w[:, :3, 1] = down
+    c2w[:, :3, 2] = forward
+    c2w[:, :3, 3] = eye_positions
+    return c2w
+
+
+def create_demo_trajectory(
+    gaussians: Gaussians3D,
+    intrinsics_px: torch.Tensor,
+    image_shape: tuple[int, int],
+) -> tuple[torch.Tensor, torch.Tensor]:
+    """Create a single-image novel-view trajectory."""
+    device = gaussians.mean_vectors.device
+    height, width = image_shape
+    focal_length_px = float(intrinsics_px[0, 0].item())
+
+    points = gaussians.mean_vectors[0].detach().to(device=device, dtype=torch.float32)
+    valid_mask = torch.isfinite(points).all(dim=-1) & (points[:, 2] > 1e-4)
+    if valid_mask.any():
+        valid_points = points[valid_mask]
+    else:
+        valid_points = points[torch.isfinite(points).all(dim=-1)]
+    if valid_points.numel() == 0:
+        raise ValueError("Could not derive a valid trajectory because all gaussian centers are invalid.")
+
+    look_at = valid_points.median(dim=0).values
+    min_depth = torch.quantile(valid_points[:, 2], 0.1).clamp_min(1e-3)
+
+    diagonal = math.sqrt((width / focal_length_px) ** 2 + (height / focal_length_px) ** 2)
+    max_lateral_offset = 0.08 * diagonal * float(min_depth.item())
+    max_medial_offset = 0.15 * float(min_depth.item())
+
+    phase = torch.linspace(
+        0.0,
+        1.0,
+        steps=DEFAULT_VIDEO_FRAMES,
+        device=device,
+        dtype=torch.float32,
+    )
+    eye_positions = torch.stack(
+        [
+            max_lateral_offset * torch.sin(2.0 * torch.pi * phase),
+            torch.zeros(DEFAULT_VIDEO_FRAMES, device=device),
+            max_medial_offset * (1.0 - torch.cos(2.0 * torch.pi * phase)) / 2.0,
+        ],
+        dim=-1,
+    )
+
+    c2w = _build_look_at_c2w(
+        eye_positions=eye_positions,
+        look_at=look_at,
+        world_up=torch.tensor([0.0, -1.0, 0.0], device=device, dtype=torch.float32),
+    )
+    extrinsics = torch.linalg.inv(c2w)
+    intrinsics_norm = normalize_intrinsics(intrinsics_px, width, height).to(device=device, dtype=torch.float32)
+    intrinsics_norm = repeat(intrinsics_norm, "i j -> 1 t i j", t=DEFAULT_VIDEO_FRAMES)
+    return extrinsics.unsqueeze(0), intrinsics_norm
+
+
+@torch.inference_mode()
+def render_novel_view_video_from_single_view(
+    decoder: Decoder,
+    gaussians: Gaussians3D,
+    render_intrinsics_px: torch.Tensor,
+    render_image_shape: tuple[int, int],
+    output_path: Path,
+) -> Path:
+    """Render an RGB novel-view video for a single-image gaussian scene."""
+    device = gaussians.mean_vectors.device
+    extrinsics, intrinsics_norm = create_demo_trajectory(
+        gaussians=gaussians,
+        intrinsics_px=render_intrinsics_px.to(device=device, dtype=torch.float32),
+        image_shape=render_image_shape,
+    )
+
+    frames: list[torch.Tensor] = []
+    num_frames = extrinsics.shape[1]
+    for start in range(0, num_frames, DEFAULT_RENDER_CHUNK_SIZE):
+        end = min(start + DEFAULT_RENDER_CHUNK_SIZE, num_frames)
+        rendered = decoder.forward(
+            gaussians=gaussians,
+            extrinsics=extrinsics[:, start:end],
+            intrinsics=intrinsics_norm[:, start:end],
+            image_shape=render_image_shape,
+        )
+        frames.extend(frame.detach().cpu() for frame in rendered[0])
+
+    save_video(frames, output_path, fps=DEFAULT_VIDEO_FPS)
+    return output_path
+
+
+def _resolve_checkpoint_path(checkpoint: str | Path) -> Path:
+    checkpoint_path = Path(checkpoint)
+    if not checkpoint_path.exists():
+        raise FileNotFoundError(f"Checkpoint not found: {checkpoint_path}")
+    if not checkpoint_path.is_file():
+        raise ValueError(f"Checkpoint path is not a file: {checkpoint_path}")
+    return checkpoint_path
+
+
+def patch_supersplat_html_auto_rotate(html_path: Path) -> None:
+    """Patch SuperSplat HTML so generated camera animation starts paused.
+
+    Args:
+        html_path: Generated SuperSplat HTML path.
+
+    Returns:
+        None.
+    """
+    source = html_path.read_text(encoding="utf-8")
+    old = "noanim: url.searchParams.has('noanim'),"
+    new = "noanim: true,"
+    if new in source:
+        return
+    if old not in source:
+        raise RuntimeError(
+            "Could not pause SuperSplat HTML animation; the viewer bundle changed."
+        )
+    html_path.write_text(source.replace(old, new, 1), encoding="utf-8")
+
+
+def patch_supersplat_html_viewer_bridge(
+    html_path: Path,
+    viewer_script_path: Path | None = None,
+) -> None:
+    """Report SuperSplat's first rendered frame to its embedding container."""
+    source = html_path.read_text(encoding="utf-8")
+    marker = "data-infinisplat-viewer-bridge"
+    first_frame_hook = "window.firstFrame?.();"
+    if marker in source:
+        return
+    hook_source = source
+    if viewer_script_path is not None:
+        hook_source += viewer_script_path.read_text(encoding="utf-8")
+    if first_frame_hook not in hook_source or "</head>" not in source:
+        raise RuntimeError(
+            "Could not install the SuperSplat viewer bridge; the viewer bundle changed."
+        )
+
+    bridge = f"""
+        <script {marker}>
+            (() => {{
+                let viewerState = "loading";
+                const setViewerState = (state) => {{
+                    if (viewerState === "ready" && state !== "ready") return;
+                    viewerState = state;
+                    const frame = window.frameElement;
+                    const host = frame?.closest("[data-infinisplat-viewer]");
+                    if (host) host.dataset.viewerState = state;
+                    window.parent.postMessage(
+                        {{ type: "infinisplat:viewer-state", state }},
+                        "*"
+                    );
+                }};
+
+                window.firstFrame = () => setViewerState("ready");
+                window.addEventListener("error", () => setViewerState("error"));
+                window.addEventListener(
+                    "unhandledrejection",
+                    () => setViewerState("error")
+                );
+                window.setTimeout(() => {{
+                    if (viewerState !== "ready") setViewerState("timeout");
+                }}, 60000);
+            }})();
+        </script>
+"""
+    html_path.write_text(
+        source.replace("</head>", f"{bridge}\n    </head>", 1),
+        encoding="utf-8",
+    )

src/model/__init__.py

@@ -0,0 +1 @@
+"""Model package for InfiniSplat."""

src/model/decoder/__init__.py

@@ -0,0 +1,23 @@
+from src.model.decoder.decoder import Decoder
+from src.model.decoder.decoder_gsplat import DecoderGsplat, DecoderGsplatCfg
+
+DECODERS = {
+    "gsplat": DecoderGsplat,
+}
+
+DecoderCfg = DecoderGsplatCfg
+
+
+def get_decoder(cfg: DecoderCfg) -> Decoder:
+    decoder = DECODERS[cfg.name]
+    decoder = decoder(cfg)
+    return decoder
+
+
+__all__ = [
+    "Decoder",
+    "DecoderCfg",
+    "DecoderGsplat",
+    "DecoderGsplatCfg",
+    "get_decoder",
+]

src/model/decoder/decoder.py

@@ -0,0 +1,28 @@
+from abc import ABC, abstractmethod
+from typing import Generic, TypeVar
+
+from jaxtyping import Float
+from torch import Tensor, nn
+
+from src.utils.gaussians import Gaussians3D
+
+
+T = TypeVar("T")
+
+
+class Decoder(nn.Module, ABC, Generic[T]):
+    cfg: T
+
+    def __init__(self, cfg: T) -> None:
+        super().__init__()
+        self.cfg = cfg
+
+    @abstractmethod
+    def forward(
+        self,
+        gaussians: Gaussians3D,
+        extrinsics: Float[Tensor, "batch view 4 4"],
+        intrinsics: Float[Tensor, "batch view 3 3"],
+        image_shape: tuple[int, int],
+    ) -> Float[Tensor, "batch view 3 height width"]:
+        raise NotImplementedError

src/model/decoder/decoder_gsplat.py

@@ -0,0 +1,98 @@
+from dataclasses import dataclass
+from typing import Literal
+
+import torch
+from einops import rearrange
+from jaxtyping import Float
+from torch import Tensor
+
+from src.model.decoder.decoder import Decoder
+from src.utils.color_space import linearRGB2sRGB
+from src.utils.gaussians import Gaussians3D
+
+try:
+    from gsplat import rasterization
+except ImportError:
+    rasterization = None
+
+
+def is_gsplat_available() -> bool:
+    """Return whether optional novel-view rendering is available."""
+    return rasterization is not None
+
+
+@dataclass
+class DecoderGsplatCfg:
+    name: Literal["gsplat"]
+    background_color: list[float]
+
+
+class DecoderGsplat(Decoder[DecoderGsplatCfg]):
+    background_color: Float[Tensor, "3"]
+
+    def __init__(
+        self,
+        cfg: DecoderGsplatCfg,
+    ) -> None:
+        super().__init__(cfg)
+        self.register_buffer(
+            "background_color",
+            torch.tensor(cfg.background_color, dtype=torch.float32),
+            persistent=False,
+        )
+
+    def forward(
+        self,
+        gaussians: Gaussians3D,
+        extrinsics: Float[Tensor, "batch view 4 4"],
+        intrinsics: Float[Tensor, "batch view 3 3"],
+        image_shape: tuple[int, int],
+    ) -> Float[Tensor, "batch view 3 height width"]:
+        if rasterization is None:
+            raise RuntimeError(
+                "Novel-view rendering requires the optional `gsplat` package."
+            )
+
+        h, w = image_shape
+
+        xyzs = gaussians.mean_vectors.float()
+        opacitys = gaussians.opacities.float()
+        rotations = gaussians.quaternions.float()
+        scales = gaussians.singular_values.float()
+        colors = gaussians.colors.float()
+        covariances = gaussians.covariances.float() if gaussians.covariances is not None else None
+
+        # The public batch interface stores extrinsics as OpenCV world-to-camera.
+        test_w2c = extrinsics.float()
+        test_intr_normalized = intrinsics.float()
+        test_intr = test_intr_normalized.clone()
+        test_intr[:, :, 0] = test_intr_normalized[:, :, 0] * w
+        test_intr[:, :, 1] = test_intr_normalized[:, :, 1] * h
+
+        rendering, alpha, _ = rasterization(
+            xyzs,
+            rotations,
+            scales,
+            opacitys,
+            colors,
+            test_w2c,
+            test_intr,
+            w,
+            h,
+            sh_degree=None,
+            render_mode="RGB",
+            packed=True,
+            covars=covariances,
+            eps2d=1e-8,
+        )
+
+        rendered_rgb = rearrange(rendering, "b v h w c -> b v c h w")
+        alpha_rgb = rearrange(alpha, "b v h w 1 -> b v 1 h w")
+
+        rendered_rgb = linearRGB2sRGB(rendered_rgb)
+
+        backgrounds = rearrange(self.background_color.to(rendered_rgb), "c -> 1 1 c 1 1")
+        rendered_rgb = rendered_rgb + backgrounds * (1.0 - alpha_rgb)
+        rendered_rgb = rendered_rgb.clamp(0.0, 1.0)
+
+        return rendered_rgb

src/model/encoder/__init__.py

@@ -0,0 +1,19 @@
+from src.model.encoder.encoder import Encoder
+from src.model.encoder.encoder_infinidepth_query import (
+    EncoderInfiniDepthQuery,
+    EncoderInfiniDepthQueryCfg,
+)
+from src.model.encoder.encoder_infinisplat import EncoderInfiniSplat, EncoderInfiniSplatCfg
+
+ENCODERS = {
+    "infinisplat": EncoderInfiniSplat,
+    "infinisplat_infinidepth": EncoderInfiniDepthQuery,
+}
+
+EncoderCfg = EncoderInfiniSplatCfg | EncoderInfiniDepthQueryCfg
+
+
+def get_encoder(cfg: EncoderCfg) -> Encoder:
+    encoder = ENCODERS[cfg.name]
+    encoder = encoder(cfg)
+    return encoder

src/model/encoder/blocks/torchhub/dinov3/.docstr.yaml

@@ -0,0 +1,6 @@
+paths:
+    - dinov3
+exclude: dinov3/tests
+skip_init: True
+skip_private: True
+fail_under: 0

src/model/encoder/blocks/torchhub/dinov3/.gitignore

@@ -0,0 +1,18 @@
+build/
+dist/
+*.egg-info/
+**/__pycache__/
+
+**/.ipynb_checkpoints
+**/.ipynb_checkpoints/**
+
+**/notebooks
+
+# Ignore shell scripts
+*.sh
+
+# Ignore swap files
+*.swp
+
+# Ignore vscode directory
+.vscode/

src/model/encoder/blocks/torchhub/dinov3/CODE_OF_CONDUCT.md

@@ -0,0 +1,80 @@
+# Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+This Code of Conduct also applies outside the project spaces when there is a
+reasonable belief that an individual's behavior may have a negative impact on
+the project or its community.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at <opensource-conduct@meta.com>. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

src/model/encoder/blocks/torchhub/dinov3/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing to DINOv3
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Meta's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Meta has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## License
+By contributing to DINOv3, you agree that your contributions will be licensed
+under the LICENSE.md file in the root directory of this source tree.

src/model/encoder/blocks/torchhub/dinov3/DATASETS.md

@@ -0,0 +1,43 @@
+# Dataset setups for DINOv3
+
+## Evaluations
+
+### Depth Estimation on NYU
+
+Create a folder to host the [NYU dataset](https://cs.nyu.edu/~fergus/datasets/nyu_depth_v2.html) for example:
+
+```
+export DEPTH_DATASETS_ROOT=${HOME}/datasets
+mkdir -p ${DEPTH_DATASETS_ROOT}/NYU
+```
+
+We use the NYU subset extracted by [BTS](https://github.com/cleinc/bts/blob/master/tensorflow/README.md) from the 120k samples of the original NYU raw dataset.
+
+#### Option 1 -- Follow BTS's instructions
+Please follow BTS instructions to create the dataset:
+- [train set](https://github.com/cleinc/bts/blob/master/tensorflow/README.md#nyu-depvh-v2)
+- [test set](https://github.com/cleinc/bts/blob/master/README.md#prepare-nyu-depth-v2-test-set).
+
+Make sure you also download the train and test splits:
+```
+wget https://github.com/cleinc/bts/blob/master/train_test_inputs/nyudepthv2_train_files_with_gt.txt -O ${DEPTH_DATASETS_ROOT}/NYU/nyu_train.txt
+wget https://github.com/cleinc/bts/blob/master/train_test_inputs/nyudepthv2_test_files_with_gt.txt -O ${DEPTH_DATASETS_ROOT}/NYU/nyu_test.txt
+```
+
+#### Option 2 (preferred) -- Download the readily availble dataset from BinsFormer
+Alternatively, one can download the dataset from the following Google Drive [link](https://drive.google.com/file/d/1xI9ksHzCC_kUz6Z4FL_b1ttgj3RVHGwW/view?usp=sharing). If the Google Drive link is not available anymore, try Option 1.
+
+Expected contents:
+- `$DEPTH_DATASETS_ROOT/NYU/basement/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/basement_0001a/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/basement_0001b/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/bathroom/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/study_room_0004/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/study_room_0005a/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/study_room_0005b/[...]`
+- `$DEPTH_DATASETS_ROOT/NYU/nyu_test.txt`
+- `$DEPTH_DATASETS_ROOT/NYU/nyu_train.txt`
+
+Note: if data is downloaded with Option 2 make sure to rename `nyu` into `NYU`.
+

src/model/encoder/blocks/torchhub/dinov3/LICENSE.md

@@ -0,0 +1,66 @@
+# DINOv3 License
+
+*Last Updated: August 19, 2025*
+
+**“Agreement”** means the terms and conditions for use, reproduction, distribution and modification of the DINO Materials set forth herein.
+
+**“DINO Materials”** means, collectively, Documentation and the models, software and algorithms, including machine-learning model code, trained model weights, inference-enabling code, training-enabling code, fine-tuning enabling code, and other elements of the foregoing distributed by Meta and made available under this Agreement.
+
+**“Documentation”** means the specifications, manuals and documentation accompanying
+DINO Materials distributed by Meta.
+
+**“Licensee”** or **“you”** means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity’s behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
+
+**“Meta”** or **“we”** means Meta Platforms Ireland Limited (if you are located in or, if you are an entity, your principal place of business is in the EEA or Switzerland) or Meta Platforms, Inc. (if you are located outside of the EEA or Switzerland).
+
+**“Sanctions”** means any economic or trade sanctions or restrictions administered or enforced by the United States (including the Office of Foreign Assets Control of the U.S. Department of the Treasury (“OFAC”), the U.S. Department of State and the U.S. Department of Commerce), the United Nations, the European Union, or the United Kingdom.
+
+**“Trade Controls”** means any of the following: Sanctions and applicable export and import controls.
+
+By clicking “I Accept” below or by using or distributing any portion or element of the DINO Materials, you agree to be bound by this Agreement.
+
+## 1. License Rights and Redistribution.
+
+a. <ins>Grant of Rights</ins>. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Meta’s intellectual property or other rights owned by Meta embodied in the DINO Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the DINO Materials.
+
+b. <ins>Redistribution and Use</ins>.
+
+i. Distribution of DINO Materials, and any derivative works thereof, are subject to the terms of this Agreement. If you distribute or make the DINO Materials, or any derivative works thereof, available to a third party, you may only do so under the terms of this Agreement and you shall provide a copy of this Agreement with any such DINO Materials.
+
+ii.  If you submit for publication the results of research you perform on, using, or otherwise in connection with DINO Materials, you must acknowledge the use of DINO Materials in your publication.
+
+iii. Your use of the DINO Materials must comply with applicable laws and regulations, including Trade Control Laws and applicable privacy and data protection laws.
+
+iv. Your use of the DINO Materials will not involve or encourage others to reverse engineer, decompile or discover the underlying components of the DINO Materials.
+
+v. You are not the target of Trade Controls and your use of DINO Materials must comply with Trade Controls. You agree not to use, or permit others to use, DINO Materials for any activities subject to the International Traffic in Arms Regulations (ITAR) or end uses prohibited by Trade Controls, including those related to military or warfare purposes, nuclear industries or applications, espionage, or the development or use of guns or illegal weapons.
+
+## 2. User Support.
+
+Your use of the DINO Materials is done at your own discretion; Meta does not process any information nor provide any service in relation to such use.  Meta is under no obligation to provide any support services for the DINO Materials. Any support provided is “as is”, “with all faults”, and without warranty of any kind.
+
+## 3. Disclaimer of Warranty.
+
+UNLESS REQUIRED BY APPLICABLE LAW, THE DINO MATERIALS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OF ANY KIND, AND META DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE DINO MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE DINO MATERIALS AND ANY OUTPUT AND RESULTS.
+
+## 4. Limitation of Liability.
+
+IN NO EVENT WILL META OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY DIRECT OR INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
+
+## 5. Intellectual Property.
+
+a. Subject to Meta’s ownership of DINO Materials and derivatives made by or for Meta, with respect to any derivative works and modifications of the DINO Materials that are made by you, as between you and Meta, you are and will be the owner of such derivative works and modifications.
+
+b. If you institute litigation or other proceedings against Meta or any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the DINO Materials, outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Meta from and against any claim by any third party arising out of or related to your use or distribution of the DINO Materials.
+
+## 6. Term and Termination.
+
+The term of this Agreement will commence upon your acceptance of this Agreement or access to the DINO Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Meta may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the DINO Materials. Sections 3, 4 and 7 shall survive the termination of this Agreement.
+
+## 7. Governing Law and Jurisdiction.
+
+This Agreement will be governed and construed under the laws of the State of California without regard to choice of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement. The courts of California shall have exclusive jurisdiction of any dispute arising out of this Agreement.
+
+## 8. Modifications and Amendments.
+
+Meta may modify this Agreement from time to time; provided that they are similar in spirit to the current version of the Agreement, but may differ in detail to address new problems or concerns. All such changes will be effective immediately. Your continued use of the DINO Materials after any modification to this Agreement constitutes your agreement to such modification. Except as provided in this Agreement, no modification or addition to any provision of this Agreement will be binding unless it is in writing and signed by an authorized representative of both you and Meta.

src/model/encoder/blocks/torchhub/dinov3/MODEL_CARD.md

@@ -0,0 +1,432 @@
+# Model Card for DINOv3
+
+DINOv3 is a family of versatile vision foundation models that outperforms the specialized state of the art across a broad range of settings, without fine-tuning. DINOv3 produces high-quality dense features that achieve outstanding performance on various vision tasks, significantly surpassing previous self- and weakly-supervised foundation models.
+
+## Model Details
+
+These are Vision Transformer and ConvNeXt models trained following the method described in the DINOv3 paper. 12 models are provided:
+
+- 10 models pretrained on web data (LVD-1689M dataset)
+  - 1 ViT-7B trained from scratch,
+  - 5 ViT-S/S+/B/L/H+ models distilled from the ViT-7B,
+  - 4 ConvNeXt-{T/S/B/L} models distilled from the ViT-7B,
+- 2 models pretrained on satellite data (SAT-493M dataset)
+  - 1 ViT-7B trained from scratch
+  - 1 ViT-L distilled from the ViT-7B
+
+
+Each Transformer-based model takes an image as input and returns a class token, patch tokens (and register tokens). These models follow a ViT architecture, with a patch size of 16. For a 224x224 image, this results in 1 class token + 4 register tokens + 196 patch tokens = 201 tokens (for DINOv2 with registers this resulted in 1 + 4 + 256 = 261 tokens).
+
+The models can accept larger images provided the image shapes are multiples of the patch size (16). If this condition is not verified, the model will crop to the closest smaller multiple of the patch size.
+
+### Model Description
+
+- **Developed by:** Meta AI
+- **Model type:** Vision Transformer, ConvNeXt
+- **License:** [DINOv3 License](https://ai.meta.com/resources/models-and-libraries/dinov3-license/)
+
+### Model Sources
+
+- **Repository:** [https://github.com/facebookresearch/dinov3](https://github.com/facebookresearch/dinov3)
+- **Paper:** [https://arxiv.org/abs/2508.10104](https://arxiv.org/abs/2508.10104)
+
+## Uses
+
+The models are vision backbones providing multi-purpose features for downstream tasks.
+
+### Direct Use
+
+The models can be used without fine-tuning, with downstream classifiers as simple as linear layers, to obtain competitive results:
+
+- on image classification, using k-NN classifiers on the class token
+- on image classification, with logistic regression classifiers applied on the class token
+- on image classification, with a linear layer applied on the class token and the average of the patch tokens
+- on image retrieval using nearest neighbors
+- on geometric and semantic 3D keypoint correspondances
+- on depth estimation, semantic segmentation, using linear layers
+- on unsupervised object discovery
+- on video segmentation tracking
+- on video classification, using a small 4-layer attentive probe
+
+### Downstream Use
+
+While fine-tuning the models can yield some gains, it is recommended to keep this option as a last resort: the frozen features are expected to provide good performance out-of-the-box.
+
+## Bias, Risks, and Limitations
+
+Compared to DINOv2 and SEERv2, DINOv3 delivers somewhat consistent performance across income categories on geographical fairness and diversity, although with a notable performance drop in the low-income bucket compared to the highest-income bucket.
+
+DINOv3 also achieves relatively good scores across different regions, improving over its predecessor DINOv2. However, a relative difference is still observed between Europe and Africa.
+
+### Recommendations
+
+Fine-tuning is expected to increase the biases in the features produced by the model as they will be tuned to the fine-tuning labels.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+```python
+import torch
+
+model = torch.hub.load(
+    repo_or_dir='facebookresearch/dinov3',
+    model='<MODEL_NAME>',
+    weights='<PATH/OR/URL/TO/CHECKPOINT>',
+)
+
+# where MODEL_NAME can be one of:
+# - dinov3_vits16
+# - dinov3_vits16plus
+# - dinov3_vitb16
+# - dinov3_vitl16
+# - dinov3_vith16plus
+# - dinov3_vit7b16
+# - dinov3_convnext_tiny
+# - dinov3_convnext_small
+# - dinov3_convnext_base
+# - dinov3_convnext_large
+
+# For instance
+dinov3_vits16 = torch.hub.load(
+    repo_or_dir='facebookresearch/dinov3',
+    model='dinov3_vits16',
+    weights='<PATH/OR/URL/TO/DINOV3/VITS16/LVD1689M/CHECKPOINT>',
+)
+```
+
+## Training Details
+
+### Training Data
+
+- Web dataset (LVD-1689M): a curated dataset of 1,689 millions of images extracted from a large data
+pool of 17 billions web images collected from public posts on Instagram
+
+- Satellite dataset (SAT-493M): a dataset of 493 millions of 512x512 images sampled randomly from Maxar RGB ortho-rectified imagery at 0.6 meter resolution
+
+### Training Procedure
+
+**Training objective:**
+
+- DINO self-distillation loss with multi-crop
+- iBOT masked-image modeling loss
+- KoLeo regularization on [CLS] tokens
+- Gram anchoring
+
+- **Training regime:** PyTorch FSDP2 (with bf16 and fp8 matrix multiplications)
+
+**Distillation:**
+
+- Distillation follows the standard DINOv3 pretraining procedure, except the teacher is a frozen pretrained ViT-7B.
+
+## Evaluation
+
+**Results**
+
+The reader is referred to the associated paper for details on the evaluation protocols
+
+*Results for ViT backbones pretrained (or distilled) on web (LVD-1689M)*
+
+<table>
+  <tr>
+    <th></th>
+    <!-- <th></th> -->
+    <th colspan="4">Global Tasks</th>
+    <th colspan="5">Dense Tasks</th>
+  </tr>
+  <tr>
+    <th>Model</th>
+    <!-- <th>Dataset</th> -->
+    <th>IN-ReaL</th>
+    <th>IN-R</th>
+    <th>Obj.Net</th>
+    <th>Ox.-H</th>
+    <th>ADE20k</th>
+    <th>NYU↓</th>
+    <th>DAVIS</th>
+    <th>NAVI</th>
+    <th>SPair</th>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-S/16</td>
+    <!-- <td>LVD-1689M</td> -->
+    <td align="right">87.0</td>
+    <td align="right">60.4</td>
+    <td align="right">50.9</td>
+    <td align="right">49.5</td>
+    <td align="right">47.0</td>
+    <td align="right">0.403</td>
+    <td align="right">72.7</td>
+    <td align="right">56.3</td>
+    <td align="right">50.4</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-S+/16</td>
+    <!-- <td>LVD-1689M</td> -->
+    <td align="right">88.0</td>
+    <td align="right">68.8</td>
+    <td align="right">54.6</td>
+    <td align="right">50.0</td>
+    <td align="right">48.8</td>
+    <td align="right">0.399</td>
+    <td align="right">75.5</td>
+    <td align="right">57.1</td>
+    <td align="right">55.2</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-B/16</td>
+    <!-- <td>LVD-1689M</td> -->
+    <td align="right">89.3</td>
+    <td align="right">76.7</td>
+    <td align="right">64.1</td>
+    <td align="right">58.5</td>
+    <td align="right">51.8</td>
+    <td align="right">0.373</td>
+    <td align="right">77.2</td>
+    <td align="right">58.8</td>
+    <td align="right">57.2</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-L/16</td>
+    <!-- <td>LVD-1689M</td> -->
+    <td align="right">90.2</td>
+    <td align="right">88.1</td>
+    <td align="right">74.8</td>
+    <td align="right">63.1</td>
+    <td align="right">54.9</td>
+    <td align="right">0.352</td>
+    <td align="right">79.9</td>
+    <td align="right">62.3</td>
+    <td align="right">61.3</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-H+/16</td>
+    <!-- <td>LVD-1689M</td> -->
+    <td align="right">90.3</td>
+    <td align="right">90.0</td>
+    <td align="right">78.6</td>
+    <td align="right">64.5</td>
+    <td align="right">54.8</td>
+    <td align="right">0.352</td>
+    <td align="right">79.3</td>
+    <td align="right">63.3</td>
+    <td align="right">56.3</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-7B/16</td>
+    <!-- <td>LVD-1689M</td> -->
+    <td align="right">90.4</td>
+    <td align="right">91.1</td>
+    <td align="right">91.1</td>
+    <td align="right">72.8</td>
+    <td align="right">55.9</td>
+    <td align="right">0.309</td>
+    <td align="right">79.7</td>
+    <td align="right">64.4</td>
+    <td align="right">58.7</td>
+  </tr>
+</table>
+
+*Results for ConvNeXt backbones distilled on web (LVD-1689M)*
+
+<table>
+  <tr>
+    <th></th>
+    <th colspan="6">Global Tasks</th>
+    <th colspan="2">Dense Tasks</th>
+  </tr>
+  <tr>
+    <th>Model</th>
+    <th colspan="2">IN-ReaL</th>
+    <th colspan="2">IN-R</th>
+    <th colspan="2">Obj.Net</th>
+    <th>ADE20k</th>
+    <th>NYU↓</th>
+  </tr>
+  <tr>
+    <td></th>
+    <td>@256px</td>
+    <td>@512px</td>
+    <td>@256px</td>
+    <td>@512px</td>
+    <td>@256px</td>
+    <td>@512px</td>
+    <td colspan="2"></td>
+  </tr>
+  <tr>
+    <td>DINOv3 ConvNeXt Tiny</td>
+    <td align="right">86.6</td>
+    <td align="right">87.7</td>
+    <td align="right">73.7</td>
+    <td align="right">74.1</td>
+    <td align="right">52.6</td>
+    <td align="right">58.7</td>
+    <td align="right">42.7</td>
+    <td align="right">0.448</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ConvNeXt Small</td>
+    <td align="right">87.9</td>
+    <td align="right">88.7</td>
+    <td align="right">73.7</td>
+    <td align="right">74.1</td>
+    <td align="right">52.6</td>
+    <td align="right">58.7</td>
+    <td align="right">44.8</td>
+    <td align="right">0.432</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ConvNeXt Base</td>
+    <td align="right">88.5</td>
+    <td align="right">89.2</td>
+    <td align="right">77.2</td>
+    <td align="right">78.2</td>
+    <td align="right">56.2</td>
+    <td align="right">61.3</td>
+    <td align="right">46.3</td>
+    <td align="right">0.420</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ConvNeXt Large</td>
+    <td align="right">88.9</td>
+    <td align="right">89.4</td>
+    <td align="right">81.3</td>
+    <td align="right">82.4</td>
+    <td align="right">59.3</td>
+    <td align="right">65.2</td>
+    <td align="right">47.8</td>
+    <td align="right">0.403</td>
+  </tr>
+</table>
+
+*Results for ViT backbones pretrained (or distilled) on satellite (SAT-493M)*
+
+<table>
+  <tr>
+    <th></th>
+    <th colspan="7">(GEO-Bench) Classification</th>
+  </tr>
+  <tr>
+    <th>Model</ht>
+    <th>m-BEnet</th>
+    <th>m-brick-kiln
+    <th>m-eurosat</th>
+    <th>m-forestnet</th>
+    <th>m-pv4ger</th>
+    <th>m-so2sat</th>
+    <th>mean</th>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-L/16</td>
+    <td>73.0</td>
+    <td>96.5</td>
+    <td>94.1</td>
+    <td>60.6</td>
+    <td>96.0</td>
+    <td>57.4</td>
+    <td>79.6</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-7B/16</td>
+    <td>74.0</td>
+    <td>97.2</td>
+    <td>94.8</td>
+    <td>62.3</td>
+    <td>96.1</td>
+    <td>62.1</td>
+    <td>81.1</td>
+  </tr>
+  <tr>
+    <th></th>
+    <th colspan="7">(GEO-Bench) Segmentation</th>
+  </tr>
+  <tr>
+    <th>Model</th>
+    <th>m-cashew</th>
+    <th>m-chesapeake</th>
+    <th>m-NeonTree</th>
+    <th>m-nz-cattle</th>
+    <th>m-pv4ger-seg</th>
+    <th>m-SA-crop</th>
+    <th>mean</th>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-L/16</td>
+    <td>94.2</td>
+    <td>75.6</td>
+    <td>61.8</td>
+    <td>83.7</td>
+    <td>95.2</td>
+    <td>36.8</td>
+    <td>74.5</td>
+  </tr>
+  <tr>
+    <td>DINOv3 ViT-7B/16</td>
+    <td>94.1</td>
+    <td>76.6</td>
+    <td>62.6</td>
+    <td>83.4</td>
+    <td>95.5</td>
+    <td>37.6</td>
+    <td>75.0</td>
+  </tr>
+</table>
+
+
+## Environmental Impact
+
+- **Hardware Type:** Nvidia H100
+- **Hours used:** 61,440 hours for ViT-7B model training
+- **Cloud Provider:** Private infrastructure
+- **Compute Region:** USA
+- **Carbon Emitted:** 18t CO2eq
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+Vision Transformer models:
+
+- ViT-S (21M parameters): patch size 16, embedding dimension 384, 4 register tokens, 6 heads, MLP FFN, RoPE
+- ViT-S+ (29M parameters): patch size 16, embedding dimension 384, 4 register tokens, 6 heads, SwiGLU FFN, RoPE
+- ViT-B (86M parameters): patch size 16, embedding dimension 768, 4 register tokens, 12 heads, MLP FFN, RoPE
+- ViT-L (300M parameters): patch size 16, embedding dimension 1024, 4 register tokens, 16 heads, MLP FFN, RoPE
+- ViT-H+ (840M parameters): patch size 16, embedding dimension 1280, 4 register tokens, 20 heads, SwiGLU FFN, RoPE
+- ViT-7B (6716M parameters): patch size 16, embedding dimension 4096, 4 register tokens, 32 heads, SwiGLU FFN, RoPE
+
+ConvNeXt models:
+
+- ConvNeXt Tiny (29M parameters)
+- ConvNeXt Small (50M parameters)
+- ConvNeXt Base (89M parameters)
+- ConvNeXt Large (198M parameters)
+
+### Compute Infrastructure
+
+#### Hardware
+
+Nvidia H100 GPUs
+
+#### Software
+
+PyTorch 2.7
+
+## More Information
+
+See the [blog post](https://ai.meta.com/blog/dinov3-self-supervised-vision-model/) and the associated [website](https://ai.meta.com/dinov3/).
+
+## Citation
+
+**BibTeX**
+
+```
+@misc{simeoni2025dinov3,
+  title={{DINOv3}},
+  author={Sim{\'e}oni, Oriane and Vo, Huy V. and Seitzer, Maximilian and Baldassarre, Federico and Oquab, Maxime and Jose, Cijo and Khalidov, Vasil and Szafraniec, Marc and Yi, Seungeun and Ramamonjisoa, Micha{\"e}l and Massa, Francisco and Haziza, Daniel and Wehrstedt, Luca and Wang, Jianyuan and Darcet, Timoth{\'e}e and Moutakanni, Th{\'e}o and Sentana, Leonel and Roberts, Claire and Vedaldi, Andrea and Tolan, Jamie and Brandt, John and Couprie, Camille and Mairal, Julien and J{\'e}gou, Herv{\'e} and Labatut, Patrick and Bojanowski, Piotr},
+  year={2025},
+  eprint={2508.10104},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV},
+  url={https://arxiv.org/abs/2508.10104},
+}
+```

src/model/encoder/blocks/torchhub/dinov3/README.md

@@ -0,0 +1,882 @@
+:new: [2026-03-10] :fire: The [Canopy Height Maps v2 (CHMv2) model](https://arxiv.org/abs/2603.06382) and inference code are now available (more details on downloading the model weights and using the code [here](#canopy-height-maps-v2-chmv2)). The model weights are also available in [Hugging Face Hub](https://huggingface.co/facebook/dinov3-vitl16-chmv2-dpt-head) and [supported](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/chmv2.md) by the Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) library. Building on our original high-resolution canopy height maps released in 2024, CHMv2 delivers substantial improvements in accuracy, detail, and global consistency by leveraging DINOv3.
+
+[2025-11-20] Distillation code and configurations for ConvNeXt backbones are now released!
+
+[2025-10-13] [Semantic segmentation](https://github.com/facebookresearch/dinov3?tab=readme-ov-file#linear-segmentation-with-data-augmentation-on-ade20k) (ADE20K) and [monocular depth estimation](https://github.com/facebookresearch/dinov3?tab=readme-ov-file#linear-depth-estimation-on-nyuv2-depth) (NYUv2-Depth) linear probing code are now released!
+
+[2025-09-17] DINOv3 backbones are now supported by the [PyTorch Image Models / timm](https://github.com/huggingface/pytorch-image-models/) library starting with version [1.0.20](https://github.com/huggingface/pytorch-image-models/releases/tag/v1.0.20)
+
+[2025-08-29] DINOv3 backbones are [supported](https://huggingface.co/docs/transformers/model_doc/dinov3) by released versions of the Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) library starting with version [4.56.0](https://github.com/huggingface/transformers/releases/tag/v4.56.0)
+
+[2025-08-14] DINOv3 backbones are now available in [Hugging Face Hub](https://huggingface.co/collections/facebook/dinov3-68924841bd6b561778e31009) and [supported](https://huggingface.co/docs/transformers/model_doc/dinov3) by the [development](https://github.com/huggingface/transformers/) version of the Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) library
+
+# DINOv3 🦖🦖🦖
+
+**[Meta AI Research, FAIR](https://ai.meta.com/research/)**
+
+Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, <br/>
+Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, <br/>
+Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, <br/>
+Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, <br/>
+Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, <br/>
+Julien Mairal, Hervé Jégou, Patrick Labatut, Piotr Bojanowski
+
+[ :scroll: [`Paper`](https://arxiv.org/abs/2508.10104)] [ :newspaper: [`Blog`](https://ai.meta.com/blog/dinov3-self-supervised-vision-model/)] [ :globe_with_meridians: [`Website`](https://ai.meta.com/dinov3/)] [ :book: [`BibTeX`](#citing-dinov3)]
+
+Reference PyTorch implementation and models for DINOv3. For details, see the **[DINOv3](https://arxiv.org/abs/2508.10104)** paper.
+
+## Overview
+
+<div align="center">
+  <img width="1364" height="1024" alt="market" src="https://github.com/user-attachments/assets/1411f491-988e-49cb-95ae-d03fe6e3c268" />
+
+  <i></em><b>High-resolution dense features.</b><br/>We visualize the cosine similarity maps obtained with DINOv3 output features<br/> between the patches marked with a red cross and all other patches.</i>
+</div>
+
+<br/>
+
+An extended family of versatile vision foundation models producing high-quality dense features and achieving outstanding performance on various vision tasks including outperforming the specialized state of the art across a broad range of settings, without fine-tuning
+
+## Pretrained models
+
+:information_source: Please follow the link provided below to get access to all the model weights: once accepted, an e-mail will be sent with the complete list of URLs pointing to all the available model weights (both backbones and adapters). These URLs can then be used to either:
+- download the model or adapter weights to a local filesystem and point `torch.hub.load()` to these local weights via the `weights` or `backbone_weights` parameters, or
+- directly invoke `torch.hub.load()` to download and load a backbone or an adapter from its URL via also the `weights` or `backbone_weights` parameters.
+
+See the example code snippets below.
+
+:warning: Please use `wget` instead of a web browser to download the weights.
+
+ViT models pretrained on web dataset (LVD-1689M):
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Model</th>
+      <th>Parameters</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-S/16 distilled </td>
+      <td align="right">21M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ViT-S+/16 distilled</td>
+      <td align="right">29M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ViT-B/16 distilled</td>
+      <td align="right">86M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ViT-L/16 distilled</td>
+      <td align="right">300M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ViT-H+/16 distilled</td>
+      <td align="right">840M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ViT-7B/16</td>
+      <td align="right">6,716M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+ConvNeXt models pretrained on web dataset (LVD-1689M):
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Model</th>
+      <th>Parameters</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ConvNeXt Tiny</td>
+      <td align="right">29M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ConvNeXt Small</td>
+      <td align="right">50M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ConvNeXt Base</td>
+      <td align="right">89M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ConvNeXt Large</td>
+      <td align="right">198M</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+ViT models pretrained on satellite dataset (SAT-493M):
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Model</th>
+      <th>Parameters</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-L/16 distilled</td>
+      <td align="right">300M</td>
+      <td align="center">SAT-493M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+    <tr>
+      <td>ViT-7B/16</td>
+      <td align="right">6,716M</td>
+      <td align="center">SAT-493M</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+
+### Pretrained backbones (via PyTorch [Hub](https://docs.pytorch.org/docs/stable/hub.html))
+
+Please follow the instructions [here](https://pytorch.org/get-started/locally/) to install PyTorch (the only required dependency for loading the model). Installing PyTorch with CUDA support is strongly recommended.
+
+```python
+import torch
+
+REPO_DIR = <PATH/TO/A/LOCAL/DIRECTORY/WHERE/THE/DINOV3/REPO/WAS/CLONED>
+
+# DINOv3 ViT models pretrained on web images
+dinov3_vits16 = torch.hub.load(REPO_DIR, 'dinov3_vits16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_vits16plus = torch.hub.load(REPO_DIR, 'dinov3_vits16plus', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_vitb16 = torch.hub.load(REPO_DIR, 'dinov3_vitb16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_vitl16 = torch.hub.load(REPO_DIR, 'dinov3_vitl16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_vith16plus = torch.hub.load(REPO_DIR, 'dinov3_vith16plus', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_vit7b16 = torch.hub.load(REPO_DIR, 'dinov3_vit7b16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+
+# DINOv3 ConvNeXt models pretrained on web images
+dinov3_convnext_tiny = torch.hub.load(REPO_DIR, 'dinov3_convnext_tiny', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_convnext_small = torch.hub.load(REPO_DIR, 'dinov3_convnext_small', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_convnext_base = torch.hub.load(REPO_DIR, 'dinov3_convnext_base', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_convnext_large = torch.hub.load(REPO_DIR, 'dinov3_convnext_large', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+
+# DINOv3 ViT models pretrained on satellite imagery
+dinov3_vitl16 = torch.hub.load(REPO_DIR, 'dinov3_vitl16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+dinov3_vit7b16 = torch.hub.load(REPO_DIR, 'dinov3_vit7b16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
+```
+
+### Pretrained backbones (via Hugging Face [Transformers](https://huggingface.co/docs/transformers/))
+
+All the backbones are available in the [DINOv3](https://huggingface.co/collections/facebook/dinov3-68924841bd6b561778e31009) collection on Hugging Face Hub and supported via the Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) library (with released packages from version 4.56.0). Please refer to the corresponding documentation for usage, but below is a short example that demonstrates how to obtain an image embedding with either [Pipeline] or the [AutoModel] class.
+
+```python
+from transformers import pipeline
+from transformers.image_utils import load_image
+
+url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
+image = load_image(url)
+
+feature_extractor = pipeline(
+    model="facebook/dinov3-convnext-tiny-pretrain-lvd1689m",
+    task="image-feature-extraction",
+)
+features = feature_extractor(image)
+```
+
+```python
+import torch
+from transformers import AutoImageProcessor, AutoModel
+from transformers.image_utils import load_image
+
+url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+image = load_image(url)
+
+pretrained_model_name = "facebook/dinov3-convnext-tiny-pretrain-lvd1689m"
+processor = AutoImageProcessor.from_pretrained(pretrained_model_name)
+model = AutoModel.from_pretrained(
+    pretrained_model_name,
+    device_map="auto",
+)
+
+inputs = processor(images=image, return_tensors="pt").to(model.device)
+with torch.inference_mode():
+    outputs = model(**inputs)
+
+pooled_output = outputs.pooler_output
+print("Pooled output shape:", pooled_output.shape)
+```
+
+where `model` and `pretrained_model_name` above can be one of:
+- `facebook/dinov3-vits16-pretrain-lvd1689m`
+- `facebook/dinov3-vits16plus-pretrain-lvd1689m`
+- `facebook/dinov3-vitb16-pretrain-lvd1689m`
+- `facebook/dinov3-vitl16-pretrain-lvd1689m`
+- `facebook/dinov3-vith16plus-pretrain-lvd1689m`
+- `facebook/dinov3-vit7b16-pretrain-lvd1689m`
+- `facebook/dinov3-convnext-base-pretrain-lvd1689m`
+- `facebook/dinov3-convnext-large-pretrain-lvd1689m`
+- `facebook/dinov3-convnext-small-pretrain-lvd1689m`
+- `facebook/dinov3-convnext-tiny-pretrain-lvd1689m`
+- `facebook/dinov3-vitl16-pretrain-sat493m`
+- `facebook/dinov3-vit7b16-pretrain-sat493m`
+
+### Image transforms
+
+For models using the LVD-1689M weights (pretrained on web images), please use the following transform (standard ImageNet evaluation transform):
+
+```python
+import torchvision
+from torchvision.transforms import v2
+
+def make_transform(resize_size: int = 256):
+    to_tensor = v2.ToImage()
+    resize = v2.Resize((resize_size, resize_size), antialias=True)
+    to_float = v2.ToDtype(torch.float32, scale=True)
+    normalize = v2.Normalize(
+        mean=(0.485, 0.456, 0.406),
+        std=(0.229, 0.224, 0.225),
+    )
+    return v2.Compose([to_tensor, resize, to_float, normalize])
+```
+
+
+For models using the SAT-493M weights (pretrained on satellite imagery), please use the following transform:
+
+
+```python
+import torchvision
+from torchvision.transforms import v2
+
+def make_transform(resize_size: int = 256):
+    to_tensor = v2.ToImage()
+    resize = v2.Resize((resize_size, resize_size), antialias=True)
+    to_float = v2.ToDtype(torch.float32, scale=True)
+    normalize = v2.Normalize(
+        mean=(0.430, 0.411, 0.296),
+        std=(0.213, 0.156, 0.143),
+    )
+    return v2.Compose([to_tensor, resize, to_float, normalize])
+```
+
+### Pretrained heads - Image classification
+
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Backbone</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Head<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-7B/16</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center">ImageNet</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+
+The (full) classifier models can be loaded via PyTorch Hub:
+
+```python
+import torch
+
+# DINOv3
+dinov3_vit7b16_lc = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_lc', source="local", weights=<DEPTHER/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+
+```
+
+### Pretrained heads - Depther trained on SYNTHMIX dataset
+
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Backbone</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Head<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-7B/16</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center">SYNTHMIX</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+
+```python
+depther = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_dd', source="local", weights=<DEPTHER/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+```
+
+Full example code of depther on an image
+
+```python
+from PIL import Image
+import torch
+from torchvision.transforms import v2
+import matplotlib.pyplot as plt
+from matplotlib import colormaps
+
+def get_img():
+    import requests
+    url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+    image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
+    return image
+
+def make_transform(resize_size: int | list[int] = 768):
+    to_tensor = v2.ToImage()
+    resize = v2.Resize((resize_size, resize_size), antialias=True)
+    to_float = v2.ToDtype(torch.float32, scale=True)
+    normalize = v2.Normalize(
+        mean=(0.485, 0.456, 0.406),
+        std=(0.229, 0.224, 0.225),
+    )
+    return v2.Compose([to_tensor, resize, to_float, normalize])
+
+depther = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_dd', source="local", weights=<DEPTHER/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+
+img_size = 1024
+img = get_img()
+transform = make_transform(img_size)
+with torch.inference_mode():
+    with torch.autocast('cuda', dtype=torch.bfloat16):
+        batch_img = transform(img)[None]
+        batch_img = batch_img
+        depths = depther(batch_img)
+
+plt.figure(figsize=(12, 6))
+plt.subplot(121)
+plt.imshow(img)
+plt.axis("off")
+plt.subplot(122)
+plt.imshow(depths[0,0].cpu(), cmap=colormaps["Spectral"])
+plt.axis("off")
+
+```
+
+#### Reproduce paper results
+
+Make sure the NYU dataset is setup following [this](DATASETS.md#depth-estimation-on-nyu).
+
+Launch the following to reproduce our paper's depth estimation results on NYUv2 with the pretrained Depther trained on SYNTHMIX:
+
+```shell
+PYTHONPATH=. python -m dinov3.run.submit dinov3/eval/depth/run.py \
+config=dinov3/eval/depth/configs/config-nyu-synthmix-dpt-inference.yaml \
+datasets.root=<PATH/TO/DATASET> \
+load_from=dinov3_vit7b16_dd \
+--output-dir <PATH/TO/OUTPUT/DIR>
+```
+
+Notes:
+- if you want to launch the code without dinov3.run.submit, you can do so using python directly or torchrun:
+
+```shell
+PYTHONPATH=. python dinov3/eval/depth/run.py \
+config=dinov3/eval/depth/configs/config-nyu-synthmix-dpt-inference.yaml \
+datasets.root=<PATH/TO/DATASET> \
+load_from=dinov3_vit7b16_dd \
+output_dir=<PATH/TO/OUTPUT/DIR>
+```
+
+- One can also save prediction results using `result_config.save_results=true`.
+
+
+### Pretrained heads - Detector trained on COCO2017 dataset
+
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Backbone</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Head<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-7B/16</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center">COCO2017</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+
+```python
+detector = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_de', source="local", weights=<DETECTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+```
+
+### Pretrained heads - Segmentor trained on ADE20K dataset
+
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th>Backbone</th>
+      <th>Pretraining<br/>Dataset</th>
+      <th>Head<br/>Dataset</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-7B/16</td>
+      <td align="center">LVD-1689M</td>
+      <td align="center">ADE20K</td>
+      <td align="center"><a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a></td>
+    </tr>
+  </tbody>
+</table>
+
+```python
+segmentor = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_ms', source="local", weights=<SEGMENTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+```
+
+Example command to run a full inference on ADE20K with the provided segmentor (ViT-7B + M2F):
+
+```shell
+PYTHONPATH=. python -m dinov3.run.submit dinov3/eval/segmentation/run.py \
+config=dinov3/eval/segmentation/configs/config-ade20k-m2f-inference.yaml  \
+datasets.root=<PATH/TO/DATASET> \
+load_from=dinov3_vit7b16_ms \
+--output-dir <PATH/TO/OUTPUT/DIR>
+```
+
+Full example code of segmentator on an image
+
+```python
+import sys
+sys.path.append(REPO_DIR)
+
+from PIL import Image
+import torch
+from torchvision import transforms
+import matplotlib.pyplot as plt
+from matplotlib import colormaps
+from functools import partial
+from dinov3.eval.segmentation.inference import make_inference
+
+
+def get_img():
+    import requests
+    url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+    image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
+    return image
+
+def make_transform(resize_size: int | list[int] = 768):
+    to_tensor = v2.ToImage()
+    resize = v2.Resize((resize_size, resize_size), antialias=True)
+    to_float = v2.ToDtype(torch.float32, scale=True)
+    normalize = v2.Normalize(
+        mean=(0.485, 0.456, 0.406),
+        std=(0.229, 0.224, 0.225),
+    )
+    return v2.Compose([to_tensor, resize, to_float, normalize])
+
+segmentor = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_ms', source="local", weights=<SEGMENTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+
+img_size = 896
+img  = get_img()
+transform = make_transform(img_size)
+with torch.inference_mode():
+    with torch.autocast('cuda', dtype=torch.bfloat16):
+        batch_img = transform(img)[None]
+        pred_vit7b = segmentor(batch_img)  # raw predictions
+        # actual segmentation map
+        segmentation_map_vit7b = make_inference(
+            batch_img,
+            segmentor,
+            inference_mode="slide",
+            decoder_head_type="m2f",
+            rescale_to=(img.size[-1], img.size[-2]),
+            n_output_channels=150,
+            crop_size=(img_size, img_size),
+            stride=(img_size, img_size),
+            output_activation=partial(torch.nn.functional.softmax, dim=1),
+        ).argmax(dim=1, keepdim=True)
+plt.figure(figsize=(12, 6))
+plt.subplot(121)
+plt.imshow(img)
+plt.axis("off")
+plt.subplot(122)
+plt.imshow(segmentation_map_vit7b[0,0].cpu(), cmap=colormaps["Spectral"])
+plt.axis("off")
+```
+
+
+
+
+### Pretrained heads - Zero-shot tasks with `dino.txt`
+
+<table style="margin: auto">
+  <thead>
+    <tr>
+      <th rowspan="2">Backbone</th>
+      <th>Download</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>ViT-L/16 distilled</td>
+      <td align="center">
+        <a href="https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/">[link]</a>,
+        <a href="https://dl.fbaipublicfiles.com/dinov3/thirdparty/bpe_simple_vocab_16e6.txt.gz">vocabulary</a>,
+        <a href="https://dl.fbaipublicfiles.com/dinov2/thirdparty/LICENSE">vocabulary license</a>
+      </td>
+    </tr>
+  </tbody>
+</table>
+
+The (full) dino.txt model can be loaded via PyTorch Hub:
+
+```python
+import torch
+# DINOv3
+dinov3_vitl16_dinotxt_tet1280d20h24l, tokenizer = torch.hub.load(REPO_DIR, 'dinov3_vitl16_dinotxt_tet1280d20h24l', weights=<SEGMENTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
+```
+
+
+## Installation
+
+The training and evaluation code requires PyTorch version >= 2.7.1 as well as a few other 3rd party packages. Note that the code has only been tested with the specified versions and also expects a Linux environment. To setup all the required dependencies for training and evaluation, please follow the instructions below:
+
+*[micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)* **(Recommended)** - Clone the repository and then create and activate a `dinov3` conda environment using the provided environment definition:
+
+```shell
+micromamba env create -f conda.yaml
+micromamba activate dinov3
+```
+
+## Getting started
+
+Several notebooks are provided to get started applying DINOv3:
+- [PCA of patch features](notebooks/pca.ipynb): display the PCA of DINOv3 patch features on a foreground object (rainbow visualizations from the paper) [[Run in Google Colab]](https://colab.research.google.com/github/facebookresearch/dinov3/blob/main/notebooks/pca.ipynb)
+- [Foreground segmentation](notebooks/foreground_segmentation.ipynb): train a linear foreground segmentation model based on DINOv3 features [[Run in Google Colab]](https://colab.research.google.com/github/facebookresearch/dinov3/blob/main/notebooks/foreground_segmentation.ipynb)
+- [Dense and sparse matching](notebooks/dense_sparse_matching.ipynb): match patches from objects on two different images based on DINOv3 features [[Run in Google Colab]](https://colab.research.google.com/github/facebookresearch/dinov3/blob/main/notebooks/dense_sparse_matching.ipynb)
+- [Segmentation tracking](notebooks/segmentation_tracking.ipynb): video segmentation tracking using a non-parametric method based on DINOv3 features [[Run in Google Colab]](https://colab.research.google.com/github/facebookresearch/dinov3/blob/main/notebooks/segmentation_tracking.ipynb)
+- [Zero-shot segmentation with DINOv3-based dino.txt](notebooks/dinotxt_segmentation_inference.ipynb): compute the open-vocabulary segmentation results with dino.txt strategy.
+
+## Data preparation
+
+### ImageNet-1k
+
+The root directory of the dataset should hold the following contents:
+
+- `<ROOT>/test/ILSVRC2012_test_00000001.JPEG`
+- `<ROOT>/test/[..]`
+- `<ROOT>/test/ILSVRC2012_test_00100000.JPEG`
+- `<ROOT>/train/n01440764/n01440764_10026.JPEG`
+- `<ROOT>/train/[...]`
+- `<ROOT>/train/n15075141/n15075141_9993.JPEG`
+- `<ROOT>/val/n01440764/ILSVRC2012_val_00000293.JPEG`
+- `<ROOT>/val/[...]`
+- `<ROOT>/val/n15075141/ILSVRC2012_val_00049174.JPEG`
+- `<ROOT>/labels.txt`
+
+The provided dataset implementation expects a few additional metadata files to be present under the extra directory:
+
+- `<EXTRA>/class-ids-TRAIN.npy`
+- `<EXTRA>/class-ids-VAL.npy`
+- `<EXTRA>/class-names-TRAIN.npy`
+- `<EXTRA>/class-names-VAL.npy`
+- `<EXTRA>/entries-TEST.npy`
+- `<EXTRA>/entries-TRAIN.npy`
+- `<EXTRA>/entries-VAL.npy`
+
+These metadata files can be generated (once) with the following lines of Python code:
+
+```python
+from dinov3.data.datasets import ImageNet
+
+for split in ImageNet.Split:
+    dataset = ImageNet(split=split, root="<ROOT>", extra="<EXTRA>")
+    dataset.dump_extra()
+```
+
+Note that the root and extra directories do not have to be distinct directories.
+
+### ImageNet-22k
+
+Please adapt the [dataset class](dinov3/data/datasets/image_net_22k.py) to match your local setup.
+
+<br />
+
+:warning: To execute the commands provided in the next sections for training and evaluation, the `dinov3` package should be included in the Python module search path, i.e. simply prefix the command to run with `PYTHONPATH=.`.
+
+## Training
+
+### Fast setup: training DINOv3 ViT-L/16 on ImageNet-1k
+
+Run DINOv3 pre-training on 4 H100-80GB nodes (32 GPUs) in a SLURM cluster environment with submitit:
+
+```shell
+ PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
+  --nodes 4 \
+  --config-file dinov3/configs/train/vitl_im1k_lin834.yaml \
+  --output-dir <PATH/TO/OUTPUT/DIR> \
+  train.dataset_path=ImageNet22k:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
+```
+Training time is approximately 14 hours and the resulting checkpoint should reach 82.0% on k-NN eval and 83.5% on linear eval.
+
+The training code saves the weights of the teacher in the eval folder every 12500 iterations for evaluation.
+
+### Exact DINOv3 setup: training DINOv3 ViT-7B/16
+
+DINOv3 ViT-7B/16 is trained on a private dataset. The training involves 3 stages:
+- Pretraining
+- Gram anchoring
+- High resolution adaptation
+
+#### Pretraining
+
+Launch DINOV3 ViT-7B/16 pretraining on 32 nodes (256 GPUs) in a SLURM cluster environment with submitit.
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
+  --nodes 32 \
+  --config-file dinov3/configs/train/dinov3_vit7b16_pretrain.yaml \
+  --output-dir <PATH/TO/OUTPUT/DIR> \
+  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
+```
+
+#### Gram anchoring
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
+  --nodes 32 \
+  --config-file dinov3/configs/train/dinov3_vit7b16_gram_anchor.yaml \
+  --output-dir <PATH/TO/OUTPUT/DIR> \
+  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
+  gram.ckpt=<PATH/TO/GRAM_TEACHER_FROM_PREVIOUS_STEP>
+```
+
+#### High-resolution adaptation
+
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
+  --nodes 32 \
+  --config-file dinov3/configs/train/dinov3_vit7b16_high_res_adapt.yaml \
+  --output-dir <PATH/TO/OUTPUT/DIR> \
+  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
+  gram.ckpt=<PATH/TO/TEACHER_FROM_GRAM> \
+  student.resume_from_teacher_chkpt=<PATH/TO/TEACHER_FROM_GRAM>
+```
+
+## Multi-distillation
+
+### Test setup:
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
+  --nodes 1 \
+  --config-file dinov3/configs/train/multi_distillation_test.yaml \
+  --output-dir <PATH/TO/OUTPUT/DIR> \
+  --multi-distillation \
+  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
+```
+
+## Evaluation
+
+The training code regularly saves the teacher weights. In order to evaluate the model, run the following evaluation on a single node:
+
+
+### Logistic regression classification on ImageNet-1k
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/eval/log_regression.py \
+  model.config_file=<PATH/TO/OUTPUT/DIR>/config.yaml \
+  model.pretrained_weights=<PATH/TO/OUTPUT/DIR>/teacher_checkpoint.pth \
+  output_dir=<PATH/TO/OUTPUT/DIR> \
+  train.dataset=ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
+  eval.test_dataset=ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
+```
+
+### k-NN classification on ImageNet-1k
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/eval/knn.py \
+  model.config_file=<PATH/TO/OUTPUT/DIR>/config.yaml \
+  model.pretrained_weights=<PATH/TO/OUTPUT/DIR>/teacher_checkpoint.pth \
+  output_dir=<PATH/TO/OUTPUT/DIR> \
+  train.dataset=ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
+  eval.test_dataset=ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
+```
+
+### Linear classification with data augmentation on ImageNet-1k
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/eval/linear.py \
+  model.config_file=<PATH/TO/OUTPUT/DIR>/config.yaml \
+  model.pretrained_weights=<PATH/TO/OUTPUT/DIR>/teacher_checkpoint.pth \
+  output_dir=<PATH/TO/OUTPUT/DIR> \
+  train.dataset=ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
+  train.val_dataset=ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
+```
+
+### Linear segmentation with data augmentation on ADE20K
+
+```shell
+PYTHONPATH=. python -m dinov3.run.submit dinov3/eval/segmentation/run.py \
+model.dino_hub=dinov3_vit7b16 \
+config=dinov3/eval/segmentation/configs/config-ade20k-linear-training.yaml \
+datasets.root=<PATH/TO/DATASET> \
+--output-dir <PATH/TO/OUTPUT/DIR>
+```
+
+After the job completes, you will find in the output path directory you specified
+- `segmentation_config.yaml` that contains the config you trained the model with;
+- `model_final.pth`, the final linear head checkpoint at the end of training; and
+- `results-semantic-segmentation.csv` with the final metrics.
+
+
+#### Linear depth estimation on NYUv2 Depth
+```shell
+PYTHONPATH=. python -m dinov3.run.submit dinov3/eval/depth/run.py \
+    model.dino_hub=dinov3_vit7b16 \
+    config=dinov3/eval/depth/configs/config-nyu.yaml \
+    datasets.root=<PATH/TO/DATASET> \
+    --output-dir <PATH/TO/OUTPUT/DIR>
+```
+
+After the job completes, you will find in the output path directory you specified
+- `depth_config.yaml` that contains the config you trained the model with;
+- `model_final.pth`, the final linear head checkpoint at the end of training; and
+- `results-depth.csv` with the final metrics.
+
+### Text alignment on DINOv3 using dino.txt
+
+Text alignment can be done following the method from `dino.txt` aka [DINOv2 Meets Text](https://arxiv.org/abs/2412.16334).
+
+```shell
+PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/eval/text/train_dinotxt.py \
+   --nodes 4 \
+  # An example config for text alignment is here: dinov3/eval/text/configs/dinov3_vitl_text.yaml \
+  trainer_config_file="<PATH/TO/DINOv3/TEXT/CONFIG>" \
+  output-dir=<PATH/TO/OUTPUT/DIR>
+```
+Launching the above trains text alignment on 4 nodes with 8 gpus each (32 gpus in total).
+Please note that the text alignment model in the DINOv3 paper was trained on a private dataset and here we have given an example config in ```dinov3/eval/text/configs/dinov3_vitl_text.yaml``` using ```CocoCaptions``` dataset for illustration purposes.
+Please adapt the provided ```CocoCaptions``` dataset class, the dataset can be found [here](https://www.kaggle.com/datasets/nikhil7280/coco-image-caption)
+
+
+## Canopy Height Maps v2 (CHMv2)
+
+John Brandt, Seungeun Yi, Jamie Tolan, Xinyuan Li, Peter Potapov, <br/>
+Jessica Ertel, Justine Spore, Huy V. Vo, Michaël Ramamonjisoa, Patrick Labatut, <br/>
+Piotr Bojanowski, Camille Couprie
+
+[ :scroll: [`Paper`](https://arxiv.org/abs/2603.06382)] [ :newspaper: [`Blog`](http://ai.meta.com/blog/world-resources-institute-dino-canopy-height-maps-v2)]
+
+### CHMv2 model loading (via PyTorch [Hub](https://docs.pytorch.org/docs/stable/hub.html))
+
+:information_source: Please follow the link provided below to get access to the CHMv2 model weights: once accepted, an e-mail will be sent with the URL pointing to the available model weights. The URL can then be used to either:
+- download the model weights to a local filesystem and point `torch.hub.load()` to these local weights via the `weights` parameters, or
+- directly invoke `torch.hub.load()` to download and load a backbone from its URL.
+
+CHMv2 uses the DINOv3 ViT-L/16 satellite as the backbone, available after requesting access [here](https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/).
+
+:warning: Please use `wget` instead of a web browser to download the weights.
+
+Download link: https://ai.meta.com/resources/models-and-libraries/chmv2-downloads/
+
+```python
+import torch
+from dinov3.hub.backbones import Weights
+
+REPO_DIR = <PATH/TO/A/LOCAL/DIRECTORY/WHERE/THE/DINOv3/REPO/WAS/CLONED>
+
+chmv2_model = torch.hub.load(
+    REPO_DIR,
+    'dinov3_vitl16_chmv2',
+    source="local",
+    weights="<CHMV2_MODEL/CHECKPOINT/URL/OR/PATH>",
+    backbone_weights=Weights.SAT493M,  # or <DINOV3_VITL_SAT/CHECKPOINT/URL/OR/PATH>
+)
+```
+
+Refer to this [notebook](notebooks/chmv2_inference.ipynb) for an example of how to use the DINOv3 + CHMv2 model.
+
+This [notebook](notebooks/chmv2_dataset_exploration.ipynb) can be used to download inference data from the existing global dataset stored on aws.
+
+### CHMv2 model loading (via Hugging Face [Transformers](https://huggingface.co/docs/transformers/))
+
+The CHMv2 model is also available on [Hugging Face Hub](https://huggingface.co/facebook/dinov3-vitl16-chmv2-dpt-head) and supported via the Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) library. Please refer to the corresponding documentation for usage, but below is a short example that demonstrates how to obtain canopy height predictions on a sample image.
+
+```python
+from PIL import Image
+import torch
+
+from transformers import AutoModelForDepthEstimation, AutoImageProcessor
+
+processor = AutoImageProcessor.from_pretrained("facebook/dinov3-vitl16-chmv2-dpt-head")
+model = AutoModelForDepthEstimation.from_pretrained("facebook/dinov3-vitl16-chmv2-dpt-head")
+
+image = Image.open("image.tif")
+inputs = processor(images=image, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+depth = processor.post_process_depth_estimation(
+    outputs, target_sizes=[(image.height, image.width)]
+)[0]["predicted_depth"]
+```
+
+## License
+
+DINOv3 code and model weights are released under the DINOv3 License. See [LICENSE.md](LICENSE.md) for additional details.
+
+## Contributing
+
+See [contributing](CONTRIBUTING.md) and the [code of conduct](CODE_OF_CONDUCT.md).
+
+## Citing DINOv3
+
+If you find this repository useful, please consider giving a star :star: and citation :t-rex::
+
+```
+@misc{simeoni2025dinov3,
+  title={{DINOv3}},
+  author={Sim{\'e}oni, Oriane and Vo, Huy V. and Seitzer, Maximilian and Baldassarre, Federico and Oquab, Maxime and Jose, Cijo and Khalidov, Vasil and Szafraniec, Marc and Yi, Seungeun and Ramamonjisoa, Micha{\"e}l and Massa, Francisco and Haziza, Daniel and Wehrstedt, Luca and Wang, Jianyuan and Darcet, Timoth{\'e}e and Moutakanni, Th{\'e}o and Sentana, Leonel and Roberts, Claire and Vedaldi, Andrea and Tolan, Jamie and Brandt, John and Couprie, Camille and Mairal, Julien and J{\'e}gou, Herv{\'e} and Labatut, Patrick and Bojanowski, Piotr},
+  year={2025},
+  eprint={2508.10104},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV},
+  url={https://arxiv.org/abs/2508.10104},
+}
+```

src/model/encoder/blocks/torchhub/dinov3/VENDORED_FROM.md

@@ -0,0 +1,9 @@
+# Vendored DINOv3
+
+- Upstream repository: https://github.com/facebookresearch/dinov3.git
+- Pinned commit: `6e50ab28b75133230b6cc3a846a8adc3c300b27d`
+- Import date: `2026-03-27`
+- Scope: `InfiniSplat image branch only`
+- Notes: Imported from the official upstream repository into this workspace snapshot. The nested `.git` directory is intentionally removed.
+- Local patch: `dinov3/hub/backbones.py` accepts local `dinov3_vitl16` checkpoint paths without the upstream `-<8char_hash>.pth` suffix and infers the correct `untie_global_and_local_cls_norm` mode from the filename.
+- Local patch: `dinov3/hub/backbones.py` loads local checkpoint paths with `torch.load(...)` directly instead of converting them to `file://` URLs and copying them into `~/.cache/torch/hub`.