ProjFlow

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 18 Open GitHub ↗

[CVPR 2026] ProjFlow: Projection Sampling with Flow Matching for Zero-Shot Exact Spatial Motion Control

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
<h1>ProjFlow: Projection Sampling with Flow Matching for Zero-Shot Exact Spatial Motion Control</h1>
<p align="center">
  <a href="https://arxiv.org/abs/2602.22742">
    <img src="https://img.shields.io/badge/arXiv-2602.22742-A42C25?style=flat&logo=arXiv" alt="arXiv">
  </a>
  <a href="https://arxiv.org/pdf/2602.22742.pdf">
    <img src="https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv" alt="paper pdf">
  </a>
  <a href="https://akihisa-watanabe.github.io/projflow.github.io/">
    <img src="https://img.shields.io/badge/Project-Page-orange?style=flat&logo=Google%20Chrome" alt="project page">
  </a>
  <a href="https://huggingface.co/Akihisa-Watanabe/ProjFlow">
    <img src="https://img.shields.io/badge/HuggingFace-Model-yellow?style=flat&logo=huggingface" alt="hugging face">
  </a>
</p>

[Akihisa Watanabe](https://akihisa-watanabe.github.io/), [Qing Yu](https://yu1ut.com/), [Edgar Simo-Serra](https://esslab.jp/~ess/), [Kent Fujiwara](https://kfworks.com/)

## Overview

Official PyTorch implementation of **ProjFlow**, a training-free projection sampler for enforcing exact (hard) spatial constraints for human motion generation with flow-matching motion priors.

## ⚙️ Getting Started

<details>
<summary><b>Installation, checkpoints, and data</b></summary>

### 0. Clone this repository
```bash
git clone https://github.com/Akihisa-Watanabe/ProjFlow.git
cd ProjFlow
````

---

### 1. Conda Environment

We provide a conda environment file for a reproducible setup.

```bash
conda env create -f environment.yml
conda activate projflow
```

---

### 2. Models and Dependencies

ProjFlow is a **sampler**: it runs on top of a pretrained motion prior (we use an ACMDM Flow backbone in the paper).
Depending on what you want to do, you may need different assets:

* **Demo / generation**: requires the pretrained motion prior checkpoint.
* **Evaluation (paper metrics)**: additionally requires standard HumanML3D evaluator checkpoints + GloVe metadata files.

#### 2.1 Download pre-trained checkpoints (required)

We host the pretrained checkpoints on Hugging Face:

* [https://huggingface.co/Akihisa-Watanabe/ProjFlow](https://huggingface.co/Akihisa-Watanabe/ProjFlow)

You can download them with the HF CLI:

```bash
# Install Hugging Face Hub CLI if needed
pip install -U "huggingface_hub[cli]"

# Download checkpoints into the current directory (keeps the repo-style folder layout)
hf download Akihisa-Watanabe/ProjFlow \
  --include "checkpoints/**" \
  --local-dir .
```

Expected default location for the demo configs:

```text
./checkpoints/t2m/ACMDM_Raw_Flow_S_PatchSize22/model/latest.tar
```

#### 2.3 Evaluator checkpoints (optional; required only for evaluation scripts)

The scripts under `evaluation_*.py` follow the standard **HumanML3D** evaluation pipeline used by the ACMDM ecosystem.
To avoid duplicating (and potentially drifting from) the canonical setup steps, please follow the evaluator download + placement instructions in the [ACMDM](https://github.com/neu-vi/ACMDM) README.

ProjFlow expects the same evaluator checkpoint layout as ACMDM. In particular, the common default paths are:

```text
./checkpoints/t2m/text_mot_match/model/finest.tar
./checkpoints/t2m/text_mot_match_clip/model/finest.tar
```

#### 2.4 GloVe metadata (optional; required only for evaluation scripts)

For evaluator-side text tokenization, we use the conventional GloVe-based vocabulary metadata files.
Again, please follow the GloVe download instructions in the [ACMDM](https://github.com/neu-vi/ACMDM) README.

ProjFlow expects the following files under:

```text
./glove/our_vab_data.npy
./glove/our_vab_words.pkl
./glove/our_vab_idx.pkl
```
---

### 3. Obtain Data (optional)

You do not need to download any dataset if you only want to generate motions from text prompts and run the demo constraints.

If you want to reproduce and evaluate our method, obtain the [HumanML3D](https://github.com/EricGuo5513/HumanML3D) dataset.

By default, our scripts expect:

```text
./datasets/HumanML3D/
  new_joints/
  texts/
  train.txt
  val.txt
  test.txt
```

</details>

## 🎬 Demo

<details>
<summary><b>Demo scripts and Blender visualization</b></summary>

### 1. Demo runner (Relative, Loop, 2D-to-3D lifting)

Run from the repository root:

```bash
# Relative joint offsets
python -m demo.run --config demo/configs/relative_offset.yaml

# Loop-closure constraints
python -m demo.run --config demo/configs/loop_closure.yaml

# 2D projection lifting
python -m demo.run --config demo/configs/lift_heart.yaml
```

### 2. Inpainting demo

Inpainting uses a dedicated script:

```bash
python demo/inpaint/run_inpaint.py \
  --text "a person walks in a circle" \
  --preset circle \
  --joint_id 0 \
  --n_frames 196 \
  --name ACMDM_Raw_Flow_S_PatchSize22 \
  --model ACMDM-Raw-Flow-S-PatchSize22 \
  --out_dir outputs/demo_inpaint_circle \
  --save_mp4
```

Reference for concrete values in this inpaint example:

| Argument     | Example value                  | Description                                                           |
| ------------ | ------------------------------ | --------------------------------------------------------------------- |
| `--model`    | `ACMDM-Raw-Flow-S-PatchSize22` | Model key in `ACMDM_models`.                                          |
| `--name`     | `ACMDM_Raw_Flow_S_PatchSize22` | Checkpoint experiment folder name.                                    |
| `--joint_id` | `0`                            | Controlled joint index (`0` = pelvis/root in T2M).                    |
| `--n_frames` | `196`                          | Generated sequence length.                                            |
| `--out_dir`  | `outputs/demo_inpaint_circle`  | Output directory for `samples_world.npy`, plots, and optional videos. |

### 3. Blender visualization (headless)

You can render generated `.npy` motions with Blender:

```bash
python -m demo.projflow_blender_viz.run_headless \
  --motion outputs/demo_inpaint_circle/samples_world.npy \
  --out outputs/demo_inpaint_circle/motion_00_blender.mp4 \
  --config demo/projflow_blender_viz/config_default.json
```

The rendered motion will look something like this:
<p align="center">
  <img src="images/blender.gif" alt="Rendered motion example">
</p>

Reference for concrete values in this Blender example:

| Argument               | Example value                                   | Description                                                  |
| ---------------------- | ----------------------------------------------- | ------------------------------------------------------------ |
| `--motion`             | `outputs/demo_lift_heart/motion_00_world.npy`   | Input motion from demo output (`(T,J,3)` world coordinates). |
| `--out`                | `outputs/demo_lift_heart/motion_00_blender.mp4` | Rendered video output path.                                  |
| `--config`             | `demo/projflow_blender_viz/config_default.json` | Render settings (engine, fps, camera, lighting, style).      |
| `--blender` (optional) | `/path/to/blender`                              | Use when Blender is not found on `PATH`.                     |

If Blender is not on `PATH`, set it with `--blender /path/to/blender` (or `BLENDER_BIN`).

> [!NOTE]
> We test our code on Blender 5.0.1.

</details>

## 📊 Evaluation

<details>
<summary><b>Evaluation setup and command examples</b></summary>

Evaluation requires:

1. HumanML3D data under `./datasets/`
2. evaluator checkpoints under `./checkpoints/t2m/text_mot_match*/...`
3. GloVe metadata under `./glove/`

For (2) and (3), please follow the [ACMDM](https://github.com/neu-vi/ACMDM) README.

Run a single controlled evaluation with:

```bash
python evaluation_ProjFlow.py \
  --name ACMDM_Raw_Flow_S_PatchSize22 \
  --model ACMDM-Raw-Flow-S-PatchSize22 \
  --dataset_name t2m \
  --dataset_dir ./datasets \
  --checkpoints_dir ./checkpoints \
  --cfg 3 \
  --index 0 \
  --intensity 100
```

Reference for concrete values in this evaluation example:

| Argument                 | Example value                                                     | Description                                                               |
| ------------------------ | ----------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `--name`                 | `ACMDM_Raw_Flow_S_PatchSize22`                                    | Checkpoint experiment folder name under `./checkpoints/<dataset_name>/`.  |
| `--model`                | `ACMDM-Raw-Flow-S-PatchSize22`                                    | Model key in `ACMDM_models` (must match checkpoint architecture).         |
| `--dataset_name`         | `t2m`                                                             | Evaluation dataset.                                                       |
| `--dataset_dir`          | `./datasets`                                                      | Root directory containing dataset files.                                  |
| `--checkpoints_dir`      | `./checkpoints`                                                   | Root directory containing checkpoints and eval logs.                      |
| `--cfg`                  | `3`                                                               | Classifier-free guidance scale used during sampling.                      |
| `--index`                | `0`                                                               | Controlled joint index (multiple values allowed, e.g. `--index 0 10 11`). |
| `--intensity`            | `100`                                                             | Constraint intensity (Appendix settings: `1 2 5 25 100`).                 |

`--name` must match your checkpoint folder:
`./checkpoints/<dataset_name>/<name>/model/latest.tar`

Common joint indices used in our evaluation:

| Index | Joint      |
| ----- | ---------- |
| `0`   | Pelvis     |
| `10`  | Left toe   |
| `11`  | Right toe  |
| `15`  | Head       |
| `20`  | Left hand  |
| `21`  | Right hand |

Evaluation logs are saved to:

```bash
./checkpoints/<dataset_name>/<name>/eval/eval.log
```

</details>

## Citation

If you find our paper or code helpful, please cite:

```bibtex
@inproceedings{watanabe2026projflow,
  title   = {ProjFlow: Projection Sampling with Flow Matching for Zero-Shot Exact Spatial Motion Control},
  author  = {Akihisa Watanabe and Qing Yu and Edgar Simo-Serra and Kent Fujiwara},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year    = {2026}
}
```

ProjFlow

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README