ActorConditionedAttentionMaps

KO | EN

gitlite — search

#javascript #typescript #llm #android #rust #deep-learning #linux #java #windows #awesome #kubernetes #vue #security #mcp

ActorConditionedAttentionMaps

(★ 29)

Training code for the ACAM action detection model.

ActorConditionedAttentionMaps Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ActorConditionedAttentionMaps

**REAL TIME DEMO CODE AVAILABLE AT**: https://github.com/oulutan/ACAM_Demo/

If you use this work, please cite our paper:

```
@inproceedings{ulutan2020actor,
  title={Actor conditioned attention maps for video action detection},
  author={Ulutan, Oytun and Rallapalli, Swati and Srivatsa, Mudhakar and Torres, Carlos and Manjunath, BS},
  booktitle={The IEEE Winter Conference on Applications of Computer Vision},
  pages={527--536},
  year={2020}
}
```

The resulting model is easily transferable to other contexts. See the real-time demo video at [real-time demo link](https://drive.google.com/file/d/1YLBRqnI2FR15NmQ-JJacMOam4vaUuIIJ/view?usp=sharing). Following is a snapshot from the video.

Also tested on videos from various sources such as surveillance cameras and autonomous driving datasets. Video available at [surveillance demo link](https://drive.google.com/file/d/1FnwicGdI94XP884dP_uISMFLIE_EZ-PW/view?usp=sharing).

# Model Weights

Initialization weights (i3d_rgb_imagenet) and trained models for AVA/JHMDB datasets are available at

https://drive.google.com/drive/folders/1VhR3tOP7Hi2_ZsLonaty-d_HEVXcwfor?usp=sharing

# Training the model on AVA

Tested on python 2.7.12.

All of my python package versions are available in the requirements.txt file. Check there for version issues

Codebase is structured on multiple directories.

`scripts`: The initial processing scripts. Downloading and processing ava

`model_training`: Includes files for training the ACAM model. Includes data loaders and training files

`data`: All dataset related files go here in their own separate subfolders

`obj_detection`: This is a codebase that focuses on running object detectors on AVA and finetuning it for AVA actors. Based on maskrcnn-benchmark from facebook. https://github.com/facebookresearch/maskrcnn-benchmark

`evaluation`: this includes the evaluation codebase for the AVA. It is a submodule from ACtivityNet and includes some custom files.

1. First of all run `source set_environment.sh` in the main folder `./ActorConditionedAttentionMaps`. This sets the environment which points to the full path of this directory in your system. This is essential as everything uses this env variable. Do this whenever you login or run any of my files.

2. Go to `scripts` and run `01_download_videos.py`. This will download the movies for AVA trainval. If you want the test files change the `split` variable in the first lines of `01_download_videos.py` and run again. Most files use similar structure so you have to run them separately for splits.

3. Go to `scripts` and run `02_crop_video_segments.py`. This will crop the movies into 2-second video segments. As the data is annotated 1fps this makes an overlap between consecutive segments. This will also generate the more compact/easily accessible `segment_annotation` files. File `02b_get_segment_annotations.py` is optional and can be used to quickly update annotations without cropping. Don't run this

4. Go to the object_detector folder and follow the readme there. It shows you how to set up the object detector and run it for AVA. optionally you can fine tune the detector for AVA or download my fine-tuned weights. When everything is set you should be running `03_keyframe_detect_objects.py` script in `ActorConditionedAttentionMaps/obj_detection/maskrcnn-benchmark/AVA_scripts/` which will generate ACAM readable object detection results.

5. At this point data is ready to be trained. However, I strongly recommend extracting tfrecord files for these videos. This helps with performance significantly especially if you are using multiple gpus (unefficient GPU loading could increase the training time to months!). At this point you should have ./data/AVA/segments which has 2 second video segments and ./data/AVA/objects_finetuned_mrcnn which has object detection results. Now run `04_convert_ava_vids_with_labels.py` in `scripts` to generate tfrecords files for each segment. Instead of a one big file, I used individual tfrecord files. This helps me to filter files on the run easily by just changing the input list during training.

6. Now the data is ready. Go to `model_training` directory and run `train_multi_gpu.py` file. This script has various arguments which can be checked in between lines 105-115. Most of them have default options except GPU selection. -g argument sets the CUDA_VISIBLE_DEVICES and masks gpus. if you only want to use GPU 0 it should be `-g 0`, if you want to use GPUs 1,2,3 then `-g 1,2,3`. If you are using it in a system with a GPU scheduler, remove this part and handle it manually.

7. Model will train for 1 epoch and then evaluate the results on the validation set. Periodically check this to see your performance on AVA. This will also generate AVA-style results which could be used for ActivityNet submissions.

# Training on other Datasets

`model_training/Datasets_JHMDB.py` file is available which was used during JHMDB dataset training. Please create a similar file for your own dataset.

ActorConditionedAttentionMaps

ActorConditionedAttentionMaps Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

ActorConditionedAttentionMaps

ActorConditionedAttentionMaps Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README