VRDFormer_VRD

KO | EN

gitlite — search

#javascript #hacktoberfest #react #go #android #rust #linux #macos #java #windows #cli #vue #pytorch #openai

(★ 17)

No description available.

VRDFormer_VRD Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VRDFormer: End-to-end Video Relation Detection with Transformers
This repository provides the implementation of the [VRDFormer: End-to-end Video Relation Detection with Transformers](https://openaccess.thecvf.com/content/CVPR2022/papers/Zheng_VRDFormer_End-to-End_Video_Visual_Relation_Detection_With_Transformers_CVPR_2022_paper.pdf) paper.
The codebase builds upon [DETR](https://github.com/facebookresearch/detr), [Deformable DETR](https://github.com/fundamentalvision/Deformable-DETR) and [TrackFormer](https://github.com/timmeinhardt/trackformer).

## Abstract

Visual relation understanding plays an essential role for holistic video understanding. Most previous works adopt a multi-stage framework for video visual relation detection (VidVRD), which cannot capture long-term spatiotemporal contexts in different stages and also suffers from inefficiency. In this paper, we propose a transformerbased framework called VRDFormer to unify these decoupling stages. Our model exploits a query-based approach to autoregressively generate relation instances. We specifically design static queries and recurrent queries to enable efficient object pair tracking with spatio-temporal contexts. The model is jointly trained with object pair detection and relation classification. Extensive experiments
on two benchmark datasets, ImageNet-VidVRD and VidOR, demonstrate the effectiveness of the proposed VRDFormer, which achieves the state-of-the-art performance on both relation detection and relation tagging tasks.

## DATA PREPARATION
We refer to our [docs/DATA.md](docs/DATA.md) for detailed installation instructions.

## Installation
We refer to our [docs/INSTALL.md](docs/INSTALL.md) for detailed installation instructions.

## Train VRDFormer on VidVRD
- Train VidVRD based on detr with 8GPUs and batchsize=32
```
sh script/stage1/train_mgpu.sh
sh script/stage2/train_mgpu.sh
```

- Train VidVRD based on deformable detr
```
sh script/stage1/train_deform_mgpu.sh
sh script/stage2/train_deform_mgpu.sh
```

## Train VRDFormer on VidVRD
Train VidOR based on detr with 8GPUs and batchsize=64
```
sh script/stage1/train_mgpu_vidor.sh
sh script/stage2/train_mgpu_vidor.sh
```

Train VidOR based on deformable detr
```
sh script/stage1/train_deform_mgpu_vidor.sh
sh script/stage2/train_deform_mgpu_vidor.sh
```

VRDFormer_VRD

VRDFormer_VRD Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

VRDFormer_VRD

VRDFormer_VRD Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README