VESPA

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #llm #machine-learning #nodejs #awesome #cli #kubernetes #awesome-list #vue #security #ios

(★ 12)

Towards un(Human)supervised Open-World Pointcloud Labeling for Autonomous Driving

VESPA Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Pseudo Label Generation for 3D Object Detection
<p align="center">
  <img src="docs/VESPAIcon.png" width="300" />
</p>

![Comparison](https://raw.githubusercontent.com/TUMFTM/VESPA/main/docs/comparisonv2.jpg)

A comprehensive framework for generating pseudo labels for 3D object detection for autonomous driving using multiple complementary approaches.

## 🎯 About This Repository

This repository implements several (state-of-the-art) pseudo-labeling approaches for 3D object detection in autonomous driving scenarios. The framework combines zero-shot object segmentation, spatial clustering, appearance-based tracking, bounding box heuristics and several other techniques to generate high-quality pseudo labels for LiDAR-based 3D object detection.

📄 **Paper**: [VESPA: Towards un (Human) supervised Open-World Pointcloud Labeling for Autonomous Driving](https://arxiv.org/abs/2507.20397)

#### Key Features:
- Multiple pseudo-labeling strategies (UNION, VESPA-Direct, VESPA-Fusion)
- Support for nuScenes dataset with different class mapping schemes (class agnostic, 3 classes or 8 classes)
- Integration with OpenPCDet framework for model training
- Evaluation pipeline with nuScenes metrics
- Rerun-based visualization tools for debugging and analysis

#### System Requirements
- Ubuntu 22.04.3 LTS (or compatible Linux distribution)
- NVIDIA GPU with CUDA 11.8+ support
- Docker and Docker Compose

## 📊 Results

The VESPA-Direct approach demonstrates competitive performance compared to previous state-of-the-art methods:

**Class-agnostic results:**
![Class-agnostic results](https://raw.githubusercontent.com/TUMFTM/VESPA/main/docs/results_1_class.png)

**3 class results:**
![3 class results](https://raw.githubusercontent.com/TUMFTM/VESPA/main/docs/results_3_class.png)

**8-class results:**
![8-class results](https://raw.githubusercontent.com/TUMFTM/VESPA/main/docs/results_8_class.png)

*Performance comparison of VESPA-Direct against previous SOTA methods on nuScenes dataset*

## 🔧 How It Works

# Pipeline Overview

The framework operates in a **two-stage pipeline**, with an initial setup step:

### ⚙️ Configuration & Dataset Setup (Preliminary)
- **Config Files**: Define pseudo-labeling parameters in the `configs/<approach>/` directory, see examples  
- **Dataset Preparation**: Ensure proper formatting and placement of raw data (e.g., nuScenes), location is defined in the config file

---

### 1. Pseudo-Label Generation
Different approaches are implemented in the `main_pseudo_*.py` scripts:

**Available Methods:**
- **UNION** (`main_pseudo_union.py`): Appearance clustering-based approach  
- **VESPA-Direct** (`main_pseudo_vlm.py`): Vision-language-model-based approach  
- **VESPA-Direct+BG** (`main_pseudo_vlm2.py`): VLM with additional background segmentation and filtering
- **VESPA-Fusion** (`main_pseudo_combined.py`): Combined VLM- + HDBSCAN-based approach  
- **VESPA-Fusion+AC** (`main_pseudo_combined2.py`): Additional object discovery with appearance clustering

---

### 2. Model Training & Evaluation
- **Training**: Use `main_model_train.py` with the generated pseudo-labels  
- **Evaluation**: Automatic evaluation of the model using nuScenes detection metrics

## 🚀 Quick Start

### 1. Environment Setup

```bash
# Clone the repository
git clone <repository-url>
cd <repository-name>

# Build and run with Docker Compose
docker-compose up -d
docker-compose exec edgar_unsupervised bash
```

### 2. Generate Pseudo Labels

Choose one of the pseudo-labeling approaches:

```bash
# UNION approach (appearance clustering)
python main_pseudo_union.py 
    --config configs/union/base_20mc.yaml \ 
    --workers 10 # Optional

# VESPA-Direct approach (VLM-based)
python main_pseudo_vlm.py 
    --config configs/vlm/p_final.yaml \ 
    --workers 10 # Optional

# VESPA-Direct + background filtering
python main_pseudo_vlm2.py 
    --config configs/vlm2/base_google_dino.yaml \ 
    --workers 10 # Optional

# VESPA-Fusion approach (combined VLM + clustering)
python main_pseudo_combined.py 
    --config configs/combined/base.yaml \ 
    --workers 10 # Optional

# VESPA-Fusion approach with appearance based object discovery
python main_pseudo_combined.py 
    --config configs/combined2/base_lrt75_c100.yaml \ 
    --workers 10 # Optional
```
Generated labels will be saved to `outs/<approach>/<config_name>/#out_labels`.

The evaluation results of the pseudo labels will be in `outs/<approach>/<config_name>/#out_eval>`

### 3. Optional: Run Visualization
You can visualize a scene with the following command.
```bash
python main_pseudo_vlm.py
    --config configs/vlm/p_final.yaml
    --visualize
    --scene scene-0061
```
`--visualize` works on all other approches as well.

### 3. Train Detection Model
To train CenterPoint using OpenPCDet on the generated labels:
```bash
python main_model_train.py
    --cfg_file configs_train/nuscenes/cbgs_voxel0075_res3d_centerpoint.yaml
    --label_file "outs/vlm/p_final/#out_labels/vlm_p_final_all_1class.json"
    --pseudo_cfg_file configs/vlm/p_final.yaml
    --extra_tag union_baseline_3class
```
Always pass the label file with `all` in it's name, although only the training scenes will be used for training.
Checkpoints, logs and evaluation will be saved to `out_train/<name_of_json>/`

## 🎨 Pseudo Label Generation Methods

The framework implements five pseudo-labeling approaches:

### 1. UNION (Appearance Clustering)

**File**: `main_pseudo_union.py`  
**Description**: Appearance clustering-based approach using cross-scene object similarity  
📖 **[Detailed Method Description](docs/union.md)**  
📖[Paper](https://arxiv.org/pdf/2405.15688)

### 2. VESPA-Direct (VLM-based)
**File**: `main_pseudo_vlm.py`  
**Description**: Vision-language model approach with Grounding SAM segmentation  
📖 **[Detailed Method Description](docs/vespa_direct.md)**  
📄 **[Paper](https://arxiv.org/abs/2507.20397)** - Main contribution of this work

### 3. VESPA-Direct+BG (VLM with Background Filtering)
**File**: `main_pseudo_vlm2.py`  
**Description**: Enhanced VLM approach with background segmentation and filtering  
📖 **[Detailed Method Description](docs/vespa_direct_bg.md)**

### 4. VESPA-Fusion (Hybrid)
**File**: `main_pseudo_combined.py`  
**Description**: Combined VLM and spatial clustering approach for improved robustness  
📖 **[Detailed Method Description](docs/vespa_fusion.md)**

### 5. VESPA-Fusion+AC (Fusion with Appearance Clustering)
**File**: `main_pseudo_combined2.py`  
**Description**: Enhanced fusion with additional appearance-based object discovery  
📖 **[Detailed Method Description](docs/vespa_fusion_ac.md)**

## 🏋️ Model Training

### Supported Architectures
- **CenterPoint**: `configs_train/cbgs_voxel0075_res3d_centerpoint.yaml`
- **TransFusion**: `configs_train/transfusion_lidar.yaml`
- **SECOND**: `configs_train/cbgs_second_multihead.yaml`

### Training Configuration
The training pipeline automatically:
- Adapts dataset configuration for pseudo class mapping
- Maps class names to model architecture
- Handles multi-class training scenarios
- Provides checkpointing and resuming capabilities

### Example Training Commands
```bash
# Train CenterPoint with UNION class-agnostic pseudo labels
python main_model_train.py \
    --cfg_file configs_train/nuscenes/cbgs_voxel0075_res3d_centerpoint.yaml \
    --label_file "outs/union/base_20mc/#out_labels/union_base_20mc_all_1class.json" \
    --pseudo_cfg_file configs/union/base_20mc.yaml \

# Train TransFusion with VESPA-Fusion 8-class pseudo labels  
python main_model_train.py \
    --cfg_file configs_train/nuscenes/transfusion_lidar.yaml \
    --label_file "outs/combined/base/#out_labels/combined_base_all_8class.json" \
    --pseudo_cfg_file configs/combined/base.yaml
```

## 📊 Evaluation

The framework provides comprehensive evaluation using **nuScenes detection metrics**:

- **mAP** (mean Average Precision): Primary detection metric
- **mATE** (mean Average Translation Error): Localization accuracy  
- **mASE** (mean Average Scale Error): Size estimation accuracy
- **mAOE** (mean Average Orientation Error): Heading accuracy
- **mAVE** (mean Average Velocity Error): Velocity estimation accuracy
- **NDS** (nuScenes Detection Score): Overall weighted score

## 🐛 Debugging & Visualization

### Rerun Visualization
The framework includes integrated **Rerun** visualization for debugging:

```bash
# Start Rerun viewer
rerun &

# Run pseudo labeling with visualization
python main_pseudo_*.py --config <config> --visualize --scene <scene_name>
```

**Visualization Features**:
- LiDAR point clouds with color-coded clusters
- 3D bounding boxes at each processing stage
- Camera images with segmentation overlays
- Tracking-based velocity vectors

## 🎓 Research Background

This research work was conducted as the **Master Thesis** of **Levente Tempfli**, under the supervision of **Esteban Rivera** at the **Department of Mobility Systems Engineering** of the **TUM School of Engineering and Design** at the **Technical University of Munich**.

For more detailed information on the method, design decisions and results please refer to the [Master Thesis PDF](docs/msc_thesis.pdf).

### Citation

If you use this work in your research, please cite our paper:

```bibtex
@article{tempfli2025vespa,
  title={VESPA: Towards un (Human) supervised Open-World Pointcloud Labeling for Autonomous Driving},
  author={Tempfli, Levente and Rivera, Esteban and Lienkamp, Markus},
  journal={arXiv preprint arXiv:2507.20397},
  year={2025}
}
```

## 📚 Acknowledgments

This work builds upon several open-source projects and research contributions:

- **OpenPCDet**: 3D object detection framework
- **nuScenes**: Large-scale autonomous driving dataset and evaluation metrics
- **Grounding DINO + SAM**: Zero-shot object detection and segmentation
- **Rerun**: Real-time visualization and debugging
- **UNION**: State-of-the-art approach for pseudo labeling of autonomous driving scenes

VESPA

VESPA Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

VESPA

VESPA Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README