TBIFormer

KO | EN

gitlite — search

#hacktoberfest #python #css #html #javascript #machine-learning #react #deep-learning #nextjs #typescript #android #opencv

★ 40 Open GitHub ↗

Official implementation of paper "Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting".

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# TBIFormer
> Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting  
> [Xiaogang Peng](https://xiaogangpeng.github.io/), Siyuan Mao, [Zizhao Wu*](http://zizhao.me)

## News
- [2023/2/28]: Our paper is accepted by CVPR 2023. Thanks to my collaborators！
- [2023/4/20]: Initial code releases.

## Abstract
Multi-person pose forecasting remains a challenging problem, especially in modeling fine-grained human body interaction in complex crowd scenarios. Existing methods typically represent the whole pose sequence as a temporal series, yet overlook interactive influences among people based on skeletal body parts. In this paper, we propose a novel Trajectory-Aware Body Interaction Transformer (TBIFormer) for multi-person pose forecasting via effectively modeling body part interactions. Specifically, we construct a Temporal Body Partition Module that transforms all the pose sequences into a Multi-Person Body-Part sequence to retain spatial and temporal information based on body semantics. Then, we devise a Social Body Interaction Self-Attention (SBI-MSA) module, utilizing the transformed sequence to learn body part dynamics for inter- and intra-individual interactions. Furthermore, different from prior Euclidean distance-based spatial encodings, we present a novel and efficient Trajectory-Aware Relative Position Encoding for SBI-MSA to offer discriminative spatial information and additional interactive clues. On both short- and long-term horizons, we empirically evaluate our framework on CMU-Mocap, MuPoTS-3D as well as synthesized datasets (6 ~ 10 persons), and demonstrate that our method greatly outperforms the state-of-the-art methods.

## Overview 
Pipeline:

Prediction Results:

<img src="./assets/results.jpg" width="800px">


<img src="./assets/figure5.png" width="720px">

## Prepare Data
We mostly follow the preprocessing procedure of [MRT](https://github.com/jiashunwang/MRT) for mixing dataset. Due to the mixing has random operations, we have uploaded the mixed dataset and others for your convenience and fair comparision. The datasets can be downloade from [![One Drive](https://img.shields.io/badge/One-Drive-blue)](https://1drv.ms/f/s!AkH2L7J4yj6XgSddjwp3QwPe0iFB?e=RYJKZm). Please prepare the data like this:
```
project_folder/
├── checkpoints/
│ ├── ...
├── data/
│ ├── Mocap_UMPM
│ │ ├── train_3_75_mocap_umpm.npy
│ │ ├── test_3_75_mocap_umpm.npy
│ │ ├── test_3_75_mocap_umpm_shuffled.npy
│ ├── MuPoTs3D
│ │ ├── mupots_150_2persons.npy
│ │ ├── mupots_150_3persons.npy
│ ├── mix1_6persons.npy
│ ├── mix2_10persons.npy
├── models/
│ ├── ...
├── utils/
│ ├── ...
├── train.py
├── test.py
```

## Training
```
python train.py
```
## Inference
```
python test.py
```

## Requirements
<a href="https://pytorch.org/get-started/locally/"><img alt="PyTorch" src="https://img.shields.io/badge/PyTorch 1.11.0+cu113-ee4c2c?logo=pytorch&logoColor=white"></a> 
<a href="https://pytorch.org/get-started/locally/"><img alt="Python" src="https://img.shields.io/badge/Python 3.7-eeeeee?logo=python&logoColor=black"></a> 
<a href="https://pytorch.org/get-started/locally/"><img alt="Python" src="https://img.shields.io/badge/torch--dct-%200.1.5-blue"> </a> 
<a href="https://pytorch.org/get-started/locally/"><img alt="Python" src="https://img.shields.io/badge/easy--dict-%201.9-blue"> </a> 
<a href="https://pytorch.org/get-started/locally/"><img alt="Python" src="https://img.shields.io/badge/numpy-%201.21.5-blue"> </a> 
<a href="https://pytorch.org/get-started/locally/"><img alt="Python" src="https://img.shields.io/badge/tqdm-%204.64.0-blue"> </a> 
<a href="https://pytorch.org/get-started/locally/"><img alt="Python" src="https://img.shields.io/badge/matplotlib-%203.2.2-blue"> </a>

## Acknowledgement
Many thanks to the previous projects:
- [MRT](https://github.com/jiashunwang/MRT)
- [HRI](https://github.com/wei-mao-2019/HisRepItself)
- [MSR](https://github.com/Droliven/MSRGCN)

Related Projects:
- [iRPE](https://github.com/microsoft/Cream/tree/main/iRPE) (orignal code of piecewise index fuction)

## Bibtex
If this work is helpful for your research, please consider citing the following BibTeX entry.

```
@InProceedings{peng2023trajectory,
        author    = {Peng, Xiaogang and Mao, Siyuan and Wu, Zizhao},
        title     = {Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting},
        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
        month     = {June},
        year      = {2023},
        pages     = {17121-17130}
}
```

## License
Our code is under MIT protocol.

TBIFormer

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README