motion-style-transfer

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

motion-style-transfer

★ 22 Open GitHub ↗

[CoRL22] Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Motion Style Transfer

Short Version (*Accepted to ICML Workshop "Safe Learning for Autonomous Driving"*)

**[`Paper`](https://learn-to-race.org/workshop-sl4ad-icml2022/assets/papers/paper_19.pdf) | [`Teaser`](https://drive.google.com/file/d/1f6E4GjVnd006udfVCrU88y0TIm12Ps8u/view?usp=sharing) | [`Video`](https://drive.google.com/file/d/1lkNXW1BbkB6cWD0-3r5gs8w-zoHNwPPj/view?usp=sharing) | [`Poster`](https://drive.google.com/file/d/10JXVbmUOo5BAxeBI4YP5EAxkoHL9AJ19/view?usp=sharing) | [`Level 5`](https://github.com/vita-epfl/motion-style-transfer-l5kit)**

**Modular Low-Rank Style Transfer for Deep Motion Forecasting**
 
<a href="https://thedebugger811.github.io">Parth Kothari*</a>, 
<a href="https://people.epfl.ch/danya.li/?lang=en">Danya Li*</a>, 
<a href="https://sites.google.com/view/yuejiangliu">Yuejiang Liu</a>,
<a href="https://people.epfl.ch/alexandre.alahi/?lang=en/">Alexandre Alahi</a>
 
École Polytechnique Fédérale de Lausanne (EPFL)

Long Version (*Accepted to CoRL 2022*)

**Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting**

## Overview

We propose efficient adaptation of deep motion forecasting models pretrained in one domain with sufficient data to new styles with *limited samples* through the following designs:

* a low-rank motion style adapter, which projects and adapts the style features at a low-dimensional bottleneck

* a modular adapter strategy, which disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers

## Setup

Use `python=3.8`.

Install PyTorch, for example using pip

```
pip install --upgrade pip
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
```

Install dependencies
```
pip install -r requirements.txt
```

## Data

We provide raw and processed SDD dataset, segmentation models for both SDD and inD images, and pretrained models for both SDD and inD experiments. You can get them via `bash scripts/get_pretrained_model_and_data.sh`.

Due to copyright, we cannot provide data for inD dataset. Please request the inD file from the authors and then use the `bash scripts/inD/processing.sh` file for preprocessing.

The data directory should have the following structure:
```
data
├── sdd
│   ├── raw/annotations/
│   ├── filter
│   |    ├── shortterm
│   |    |    ├── avg_vel/
│   |    |    └── agent_type/
│   |    └── longterm/agent_type/
│   └── sdd_segmentation.pth 
|
└── inD-dataset-v1.0
    ├── images/
    ├── data/
    ├── filter
    |    ├── shortterm/agent_type/
    |    └── longterm/agent_type/
    └── inD_segmentation.pth

```

In addition to our provided datasets, you can refer to `utils.inD_dataset.py` or `utils.sdd_dataset.py` to create customized ones.

## Experiments

There are 5 sets of experiments provided in our paper:

| Experiment | Dataset used | Model used | Prediction | Pretraining | Adaptation |
|----------|--------|----------|--------|----------|--------|
| Motion Style Transfer across Agents | SDD | Y-Net | short-term | pedestrians (all scenes) | bikers (deathCircle_0) |
| Motion Style Transfer across Scenes | inD | Y-Net | long-term | pedestrians (scenes=2,3,4) | pedestrians (scenes=1) | 
| Modular Style Adapter (Agent motion) | inD | Y-Net-Mod | long-term |cars (scenes=1) | trucks (scenes=1) | 
| Modular Style Adapter (Scene) | inD | Y-Net-Mod | short-term | pedestrians (scenes=2,3,4) | pedestrians (scenes=1) |
| Modular Style Adapter (Agent motion) | SDD | Y-Net-Mod | shot-term | bikers (low speed) | bikers (high speed)

The above experiments correpond to scripts as follows:

| Experiment | Script folder | 
|------------|---------------|
| Motion Style Transfer across Agents | `scripts/sdd/ped_to_biker/` | 
| Motion Style Transfer across Scenes | `scripts/inD/ped2ped_scene234to1/ynet/` | 
| Modular Style Adapter (Agent motion) | `scripts/inD/scene1_car_to_truck/` | 
| Modular Style Adapter (Scene) | `scripts/inD/ped2ped_scene234to1/ynetmod/` | 
| Modular Style Adapter (Agent motion) | `scripts/sdd/biker_low_to_high/` |

In each folder, scripts for pretraininng, generalizing, baseline fine-tuning, and MoSA fine-tuning are provided.

## Qualitative Results

We present three examples from motion style transfer across agents (pedestrians to bikers) on SDD. Each of the figures below illustrates the difference in goal decoder output of Y-Net on the adaptation of pedestrian-trained model using our proposed style-injection modules, where red means increased focus and blue means decreased focus. 
Adapted Y-Net successfully learns the style differences between the behavior of pedestrians and bikers:
* it correctly infers valid areas of traversal (move on bikerways instead of sidewalks)
* it updates the motion style parameters (new goal position is farther than old one)
* it effectively captures bikers' multimodality

motion-style-transfer

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README