---
license: afl-3.0
arxiv: 2507.19878
pipeline_tag: robotics
library_name: pytorch
tags:
  - robotics
  - drones
  - visual-servoing
  - knowledge-distillation
  - computer-vision
  - quadrotor
  - real-time
  - yolo
  - pytorch
language:
  - en
datasets:
  - brittleru/nser-ibvs-mask-splitter-dataset
  - custom
metrics:
  - mean_error
  - iou
model-index:
  - name: NSER-IBVS Student Network
    results:
      - task:
          type: visual-servoing
          name: Visual Servoing
        dataset:
          type: custom
          name: NSER-IBVS Simulation Dataset
        metrics:
          - type: fps
            name: Inference Speed
            value: 540.8
          - type: mean_error
            name: Mean Error (px)
            value: 14.26
          - type: iou
            name: IoU
            value: 0.752
      - task:
          type: visual-servoing
          name: Visual Servoing
        dataset:
          type: custom
          name: NSER-IBVS Real-World Dataset
        metrics:
          - type: mean_error
            name: Mean Error (px)
            value: 33.33
          - type: iou
            name: IoU
            value: 0.591
---


<div align="center">

<h1>NSER-IBVS: Efficient Self-Supervised Neuro-Analytic Visual Servoing for Real-time Quadrotor Control</h1>


**ICCV 2025 Workshops - Oral Presentation**

[Sebastian Mocanu](https://sebastianmocanu.com/) · [Sebastian-Ion Nae](https://scholar.google.com/citations?user=VBogtPAAAAAJ&hl=en) · [Mihai-Eugen Barbu](https://scholar.google.com/citations?user=9GlsUHAAAAAJ&hl=en) · [Marius Leordeanu](https://scholar.google.com/citations?user=se9kni0AAAAJ&hl=en)

[![Paper](https://img.shields.io/badge/Paper-ICCV%202025-blue)](https://openaccess.thecvf.com/content/ICCV2025W/EVW/papers/Mocanu_Efficient_Self-Supervised_Neuro-Analytic_Visual_Servoing_for_Real-time_Quadrotor_Control_ICCVW_2025_paper.pdf)
[![arXiv](https://img.shields.io/badge/arXiv-2507.19878-b31b1b)](https://arxiv.org/abs/2507.19878)
[![GitHub](https://img.shields.io/badge/GitHub-Code-black?logo=github)](https://github.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone)
[![Website](https://img.shields.io/badge/Project-Website-green)](https://spacetime-vision-robotics-laboratory.github.io/NSER-IBVS/)

</div>

---

## Model Description

This repository contains pre-trained models for **NSER-IBVS** (Numerically Stable Efficient Reduced Image-Based Visual Servoing), a self-supervised framework for vision-based quadrotor control.

The system uses a **teacher-student architecture** where a compact 1.7M parameter student network learns to control a drone by imitating an analytical IBVS teacher, achieving **11x faster inference** (540 FPS vs 48 FPS) while maintaining comparable accuracy.

**Trajectories teacher vs student from two starting points:**
<table align="center">
  <tr>
    <td align="center"><img src="https://raw.githubusercontent.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone/main/docs/images/Trajectory-Front.png" width="1200" alt="Front View Trajectories Teacher vs Student"><br>Front View Trajectories</td>
    <td align="center"><img src="https://raw.githubusercontent.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone/main/docs/images/Trajectory-Up.png" width="1200" alt="Up View Trajectories Teacher vs Student"><br>Up View Trajectories</td>
  </tr>
</table>


---

## Intended Use

- **Primary use**: Vision-based autonomous drone control for target tracking
- **Target users**: Robotics researchers, drone developers, computer vision practitioners
- **Out of scope**: Production deployment without proper safety validation

### Limitations

- Trained primarily on vehicle targets. May require fine-tuning for other objects
- Performance degrades in extreme lighting conditions or heavy occlusion
- Tested only with Parrot Anafi 4K drone (models and systems outputs piloting commands for Parrot drones).

---

## Models

### Teacher Pipeline Models

| Model                       | File                                                                        | Parameters | Description                                    |
|-----------------------------|-----------------------------------------------------------------------------|------------|------------------------------------------------|
| YOLOv11 Segmentation (sim)  | `29_05_best__yolo11n-seg_sim_car_bunker__all.pt`                            | 2.84M      | Vehicle segmentation for simulator             |
| YOLOv11 Segmentation (real) | `real-yolo-car-full-segmentation.pt`                                        | 2.84M      | Vehicle segmentation for real-world            |
| Mask Splitter (sim)         | `mask_splitter-epoch_10-dropout_0-low_x2-and-high_x0_quality_early_stop.pt` | 1.94M      | Anterior-posterior mask splitting (simulator)  |
| Mask Splitter (real)        | `mask_splitter-epoch_10-dropout_0-_x2_real_early_stop.pt`                   | 1.94M      | Anterior-posterior mask splitting (real-world) |

### Student Models

| Model                | File                                               | Parameters | Description                               |
|----------------------|----------------------------------------------------|------------|-------------------------------------------|
| Student (sim→real)   | `student_model_sim_on_real_world_distribution.pth` | 1.7M       | Trained on sim, normalized for real-world |
| Student (fine-tuned) | `student_real_pretrained_augX3_80_runs.pth`        | 1.7M       | Fine-tuned on real-world data             |

---

## Usage

### Installation

```bash
git clone --recursive https://github.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone.git
cd nser-ibvs-drone

python3 -m venv ./venv
source venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .
```

Run Tests to verify imports and functionality:
```bash
python -m unittest discover ./tests
```

### Loading Models

#### YOLO Segmentation:
```python
from mask_splitter.yolo_model import YoloSegmentation

# Initialize YOLO model
yolo = YoloSegmentation(
    model_path="real-yolo-car-full-segmentation.pt",
    confidence_threshold=0.7
)

# Segment an image
annotated_frame, binary_mask = yolo.segment_image(frame)

# Get detection info
results = yolo.detect(frame)
target = yolo.find_best_target_box(results)
print(f"Confidence: {target.confidence}, Center: {target.center}")
```

#### Mask-Splitter Network
```python
import cv2
from mask_splitter.nn.infer import MaskSplitterInference

# Initialize the model
splitter = MaskSplitterInference(
    model_path="mask_splitter-epoch_10-dropout_0-_x2_real_early_stop.pt",
    device="cuda",
    image_size=(360, 640),
    confidence_threshold=0.5
)

# Load image and mask
image = cv2.imread("frame-path.png")
mask = cv2.imread("mask-path.png", cv2.IMREAD_GRAYSCALE)

# Run inference
front_mask, back_mask = splitter.infer(image, mask)

# Visualize results
splitter.visualize(image, front_mask, back_mask)
```

#### Student Network

**Direct usage:**
```python
import torch
from nser_ibvs_drone.distiled_network.drone_command_regressor import DroneCommandRegressor

# Load model
model = DroneCommandRegressor()
model.load_model("student_model_sim_on_real_world_distribution.pth")
model.eval()

# Input: RGB image tensor [B, 3, H, W]
# Output: velocity commands [vx, vy, vyaw]
```

**Or with Student Engine:**

```python
import cv2
from nser_ibvs_drone.distiled_network.distil_engine import StudentEngine

student_model_path = "student_model_sim_on_real_world_distribution.pth"
model_engine = StudentEngine(student_model_path)

frame = cv2.imread("frame-path.png")
commands = model_engine.predict(frame)
```


For complete inference pipelines and running on algorithms on drones, see the docs in [GitHub repository](https://github.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone).

---

## Performance

| Metric              | Teacher (NSER-IBVS) | Student Network |
|---------------------|---------------------|-----------------|
| **Inference Speed** | 48.3 FPS            | **540.8 FPS**   |
| **Parameters**      | 4.78M               | **1.7M**        |
| Mean Error (Sim)    | 29.76 px            | **14.26 px**    |
| IoU (Sim)           | 0.522               | **0.752**       |
| Mean Error (Real)   | **29.96 px**        | 33.33 px        |
| IoU (Real)          | **0.627**           | 0.591           |


**Control commands and error evolutions over time:**
<table align="center">
  <tr>
    <td align="center"><img src="https://raw.githubusercontent.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone/main/docs/images/Real-IBVS-Evolution-Errors.png" width="4772" alt="Real-World Flight - Teacher (IBVS)"><br>Real-World Flight - Teacher (IBVS)</td>
    <td align="center"><img src="https://raw.githubusercontent.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone/main/docs/images/Real-Student-Evolution-Errors.png" width="4772" alt="Real-World Flight - Student"><br>Real-World Flight - Student</td>
  </tr>
</table>

<table align="center">
  <tr>
    <td align="center"><img src="https://raw.githubusercontent.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone/main/docs/images/Sim-IBVS-Evolution-Errors.png" width="4764" alt="Digital-Twin Flight - Teacher (IBVS)"><br>Digital-Twin Flight - Teacher (IBVS)</td>
    <td align="center"><img src="https://raw.githubusercontent.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone/main/docs/images/Sim-Student-Evolution-Errors.png" width="4764" alt="Digital-Twin Flight - Student"><br>Digital-Twin Flight - Student</td>
  </tr>
</table>

---

## Training Data

Training data and the custom UE4 simulator environment are available:

- [Training Data Mask-Splitter Hugging Faces brittleru/nser-ibvs-mask-splitter-dataset](https://huggingface.co/datasets/brittleru/nser-ibvs-mask-splitter-dataset)
- [Training Data (Google Drive)](https://drive.google.com/drive/folders/1T2Uq8nPQoy_QNiI87xBUoXlPAk7k-9Cq?usp=sharing)
- [UE4 Bunker Environment](https://drive.google.com/file/d/1kHqJtTq7CGoazUUn8tPFnijV3lYY4toO/view?usp=drive_link)

---

## Citation

```bibtex
@InProceedings{Mocanu_2025_ICCV,
    author    = {Mocanu, Sebastian and Nae, Sebastian-Ion and Barbu, Mihai-Eugen and Leordeanu, Marius},
    title     = {Efficient Self-Supervised Neuro-Analytic Visual Servoing for Real-time Quadrotor Control},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
    month     = {October},
    year      = {2025},
    pages     = {1744-1753}
}
```

---

## License

[Academic Free License v3.0](https://opensource.org/license/afl-3-0-php)

---

## Links

| Resource | Link                                                                                                                                                                                                     |
|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Paper    | [ICCV 2025 Open Access](https://openaccess.thecvf.com/content/ICCV2025W/EVW/papers/Mocanu_Efficient_Self-Supervised_Neuro-Analytic_Visual_Servoing_for_Real-time_Quadrotor_Control_ICCVW_2025_paper.pdf) |
| arXiv    | [2507.19878](https://arxiv.org/abs/2507.19878)                                                                                                                                                           |
| Code     | [GitHub](https://github.com/SpaceTime-Vision-Robotics-Laboratory/nser-ibvs-drone)                                                                                                                        |
| Website  | [Project Page](https://spacetime-vision-robotics-laboratory.github.io/NSER-IBVS/)                                                                                                                        |
| Poster   | [PDF](https://spacetime-vision-robotics-laboratory.github.io/NSER-IBVS/assets/NSER-IBVS-Poster-ICCV-2025.pdf)                                                                                            |