Paxion

KO | EN

gitlite — search

#hacktoberfest #ai #llm #golang #go #machine-learning #linux #docker #macos #java #awesome #kubernetes #vue #ios

(★ 38)

Repo for paper: "Paxion: Patching Action Knowledge in Video-Language Foundation Models" Neurips 23 Spotlight

Paxion Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Repo for paper: [Paxion: Patching Action Knowledge in Video-Language Foundation Models](https://arxiv.org/abs/2305.10683)

<div align="center">
    <figure>
    <img src="ActionBench_overview.png" alt="alt_text" width="100%">
    <figcaption>Overview of the Action Dynamics Benchmark (ActionBench)</figcaption>
    </figure>
</div>
<br/>
<div align="center">
    <figure>
    <img src="Paxion_overview.png" alt="alt_text" width="100%">
    <figcaption>Overview of the Paxion framework</figcaption>
    </figure>
</div>

---

### 📝 A note regarding the naming convention in this repo: we use "PatchAndFuse" or "patch_and_fuse" as an alternative name for the "Paxion" model.

# Setup
## Environment Setup 
- after cloning this repo, setup submodules by:
    ```
    git submodule update --init --recursive
    ```
    
- Setup conda environment
    ```
    conda env create -f environment.yml
    conda activate paxion
    ```

- install LAVIS library from source
    ```
    cd src/LAVIS
    pip install -e .
    ```

## Dataset Setup
### Download Annotations
- Download the annotations for **ActionBench** [here](https://uofi.box.com/s/alsp793jblpj4rpdbyt148zfk65iuqv1); and put under `ActionBench/ego4d` and `ActionBench/ssv2`
- Download the annotations for **downstream tasks** [here](https://uofi.box.com/s/sgiqjerp7ju7x8fb6c4cnx4dnej3hmiv); and put the downloaded folder under the root directory of this repo as `datasets/`

- **Annotation details** for each dataset can be found in the `.md` files in [dataset_cards](./dataset_cards/).

### Download Videos & Preprocessing
Please refer to the `.md` files in [dataset_cards](./dataset_cards/) for instructions on downloading the raw videos and preprocessing.

## Download Pretrained Backbone Checkpoints
- **InternVideo**: Download the `InternVideo-MM-L-14` checkpoint following the instructions [here](https://github.com/OpenGVLab/InternVideo#model-zoo); put the downloaded `InternVideo-MM-L-14.ckpt` under `src/pretrained_ckpt/InternVideo/InternVideo-MM-L-14.ckpt`
- **ClipViP**: Download the `CLIP-ViP-B/32` checkpoint following the instructions [here](https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP#getting-started); put the downloaded `pretrain_clipvip_base_32.pt` under `src/pretrained_ckpt/ClipViP/pretrain_clipvip_base_32.pt`
- **Singularity**: Download the `Pre-trained checkpoints` following the instructions [here](https://github.com/jayleicn/singularity#download); put the `singularity_temporal_17m` under `src/pretrained_ckpt/Singularity/singularity_temporal_17m.pth`

## Download Trained Knowledge Patcher and Knowledge Fuser
- Download the Knowledge Patcher checkpoints on actionbench [here](https://uofi.box.com/s/148d5rijmpud7pl00tknmgb3mf5ef3zv); and put under `src/pretrained_ckpt/PatchAndFuse/ActionBench`
- Download the Patch & Fuse checkpoints on downstream tasks [here](https://uofi.box.com/s/aayspmcn0yueqvtdtewwas3roejbxsud); and put under `src/pretrained_ckpt/PatchAndFuse/downstream_tasks`

# Quick Start
**[demo.py](./src/demo.py)** shows an usage example on loading a trained PatchAndFuse model (with InternVideo backbone and trained on SSv2-label) and perform inference on video-text matching.

# Code Description
We build our codebase on top of [LAVIS framework](https://github.com/salesforce/LAVIS/tree/main). Please refer to the [documentation](https://opensource.salesforce.com/LAVIS//latest/intro.html#library-design) to get an idea of the overall structure (Tasks, Models, Runners, etc). Config files for running any tasks can be found in `src/configs/projects`, we include detailed comments in the `.yaml` files for more fine-grained customization on the experimental configs.

## Training
To train Knowledge Patcher on ActionBench or further train Knowledge Fuser on downstream tasks, we provide configs under `src/configs/projects/train`; Please make sure to look into the configs and do necessary modifications (e.g. specify trained checkpoints which are marked as `#TODO`).

Here is an example for using the training configs (`run_scripts/train.sh`):
    ```
        cd src/
        bash run_scripts/train.sh
    ```

## Evaluation
To evaluate Knowledge Patcher on ActionBench or evaluate trained Knowledge Fuser on downstream tasks, we provide configs under `src/configs/projects/eval`; Please make sure to look into the configs and do necessary modifications (e.g. specify trained checkpoints which are marked as `#TODO`).

Here are two examples for using the evaluation configs (`run_scripts/eval_actionbench.sh` and `run_scripts/eval_downstream_task.sh` ):
- eval actionbench:
    ```
        cd src/
        bash run_scripts/eval_actionbench.sh 
    ```
- eval downstream task:
    ```
        cd src/
        bash run_scripts/eval_downstream_task.sh 
    ```

## Inference
For customized inference, i.e., evaluate on some specific samples and visualize the results, we provide [inference.py](./src/inference.py) where one can set spefic instance ids in the `__main__` function.

Examples for running inference on different tasks can be found in `run_scripts/inference.sh`

# Citation
```
@article{wang2023paxion,
  title={Paxion: Patching Action Knowledge in Video-Language Foundation Models},
  author={Wang, Zhenhailong and Blume, Ansel and Li, Sha and Liu, Genglin and Cho, Jaemin and Tang, Zineng and Bansal, Mohit and Ji, Heng},
  journal={arXiv preprint arXiv:2305.10683},
  year={2023}
}
```

# Acknowledgement
This code used resources from [LAVIS](https://github.com/salesforce/LAVIS), [InternVideo](https://github.com/OpenGVLab/InternVideo), [ClipViP](https://github.com/microsoft/XPretrain/tree/main/CLIP-ViP), [Singularity](https://github.com/jayleicn/singularity), and [flamingo-pytorch](https://github.com/lucidrains/flamingo-pytorch). The code is implemented using PyTorch. We thank the authors for their great work and open-sourcing the code.

Paxion

Paxion Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Paxion

Paxion Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README