Evita

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 17 Open GitHub ↗

Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing</h1>

<big>
 Chenxu Peng &nbsp;&nbsp;
 Chongtian Zhou &nbsp;&nbsp;
 Dicheng Liu &nbsp;&nbsp;
 Bowen Yin
 
 Yimian Dai &nbsp;&nbsp;
 Xialei Liu &nbsp;&nbsp;
 Ming-Ming Cheng &nbsp;&nbsp;
 Xiang Li*
 </big>


 NKIARI, Shenzhen Futian &nbsp;&nbsp;
 VCIP, CS, Nankai University &nbsp;&nbsp;
 AAIS, Nankai University

[![HomePage](https://img.shields.io/badge/-HomePage-4c8eda.svg)](https://chaineypung.github.io/evita)
[![paper](https://img.shields.io/badge/arXiv-2607.09143-brown.svg?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2607.09143)
<a href='https://huggingface.co/chainey0521/Evita/tree/main'><img src='https://img.shields.io/badge/🤗-Model-blue.svg'></a>
<a href='https://zhuanlan.zhihu.com/p/2060033757017592388'><img src='https://img.shields.io/badge/Zhihu-chinese_article-blue.svg?logo=zhihu&logoColor=white'></a>
<a href="https://pytorch.org/"><img src="https://img.shields.io/badge/Framework-PyTorch-orange.svg" /></a>
<a href="https://github.com/jamycheung/DELIVER/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" /></a>

</div>

TLDR: We propose **Evita**, a unified RGB-E backbone specifically tailored for dense parsing. With our **N-ImageNetV2** and **custom pretraining**, Evita mitigates harmonic-geometric discrepancies and adapts to diverse event formats for an optimal accuracy-latency balance.

## Contents

- [Highlights](#highlights)
- [Evita](#Evita)
- [Visualization](#visualization)
- [News](#news)
- [Installation](#installation)
- [Pretrained Weights](#pretrained-weights)
- [DeLiVER Training and Evaluation](#deliver-training-and-evaluation)
- [DDD17 and DSEC Training and Evaluation](#ddd17-and-dsec-training-and-evaluation)
- [Citation](#citation)

## Highlights

- **Unified architecture.** We build the first unified backbone specifically designed for RGB-E tasks.
- **N-ImageNetV2.** We address the large-scale misalignment between RGB and event images in N-ImageNet.
- **Flexible training.** Evita supports a "multi-representation pretraining, single-representation fine-tuning" mechanism.
- **Efficient inference.** Evita discards redundant two-stream designs and enables lightweight, efficient, end-to-end inference.

## Evita

- Overview of the proposed Evita. (a) The overall symmetric intertwined hierarchical backbone. (b) The detailed structure of the Evita block, enabling joint modality optimization. (c) The parallel routing mechanism integrating (d) Geometric Parallax Rectification and (e) Harmonic Spectral Resonance to achieve robust cross-modal feature alignment and structural frequency injection. The Transient Global Routing layer effectively captures dynamic long-range dependencies by leveraging these structurally-aligned features as kinematic anchors.

## Visualization

- Qualitative comparison on the DELIVER dataset under diverse adverse weather conditions. \texttt{Evita} exhibits superior robustness in reconstructing fine-grained structures and maintaining semantic consistency, particularly in low-light and high-occlusion scenarios, outperforming competing architectures.

## News

- [x] Release training and evaluation code.
- [x] Release DeLiVER dataset weights.
- [x] Release DDD17 and DSEC dataset weights.
- [x] Release Evita pretrained weights.
- [ ] Release N-ImageNetV2 dataset.
- [ ] Release Evita pretraining code.

## Installation

```bash
pip install -r requirements.txt
```

## Repository Structure

```text
Evita/
├── code/
│   ├── DELIVER/       # DeLiVER training and evaluation code
│   └── DDD17_DSEC/    # DDD17 and DSEC training and evaluation code
├── figs/              # Figures used in the paper and README
├── requirements.txt
└── README.md
```

## Pretrained Weights

## DeLiVER Training and Evaluation

### Data Preparation

Please prepare the DeLiVER dataset following the official instructions:

```text
https://github.com/InSAI-Lab/DELIVER/tree/main
```

### Training

Before training, please download the pretrained Evita weights.

```bash
cd path/to/DELIVER
export PYTHONPATH="path/to/DELIVER"
python -m torch.distributed.launch --nproc_per_node=4 --use_env tools/train_mm.py --cfg configs/deliver_rgbdel.yaml
```

### Evaluation

```bash
cd path/to/DELIVER
export PYTHONPATH="path/to/DELIVER"
CUDA_VISIBLE_DEVICES=0 python tools/val_mm.py --cfg configs/deliver_rgbdel.yaml
```

### DeLiVER Model Zoo

## DDD17 and DSEC Training and Evaluation

### Data Preparation

Please prepare DDD17 and DSEC following:

```text
https://github.com/CQU-UISC/MambaSeg
```

### Training

Before training, please download the pretrained Evita weights.

```bash
python train_ddd17.py
python train_dsec.py
```

### Evaluation

```bash
python evaluate.py
```

### DDD17 Model Zoo

### DSEC Model Zoo

## Acknowledgements

This repository builds upon the following excellent public repositories:

- [DELIVER](https://github.com/InSAI-Lab/DELIVER/tree/main)
- [MambaSeg](https://github.com/CQU-UISC/MambaSeg)

## License

This repository is released under the Apache-2.0 license. For commercial use, please contact the authors.

## Citation

If you use the Evita model or the N-ImageNetV2 dataset, please cite the following work:

```bibtex
@inproceedings{peng2026weaving,
  title={Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing},
  author={Chenxu Peng, Chongtian Zhou, Dicheng Liu, Bo-Wen Yin, Yimian Dai, Xialei Liu, Ming-Ming Cheng, Xiang Li},
  booktitle={Arxiv},
  year={2026}
}
```

Evita

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README