VERA - GitLite

KO | EN

gitlite — search

#javascript #python #react #ai #typescript #golang #rust #linux #nodejs #macos #kubernetes #vue #security #openai

VERA

(★ 86)

This is the code repo for the paper VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models (CVPR 2025).

파일 탐색기

최종 버전 다운로드 (.zip)

- name_gt_pair.json
- README.md
- UCF_Eval.json
- UCF_Instruct_train.json
- UCF_Instruct_val.json
- xd_annotations.json
- xd_test_800.json
- xd_train.json
- xd_val.json
- video_instruction_dataset_VERA.py
- video_instruction_dataset.py
- video_instruction_dataset_inference.py
- compute_auc.py
- generate_intial_score.py
- extract_frames.py
- extract_frames.sh
compute_AP_XD_Violence.py
compute_auc.py
compute_auc_XD_Violence.py
generate_initial_score.py
README.md
training.py
VERA_learner_instruct.txt
VERA_optimizer_instruct.txt
vera_qwen_generate_initial_score.py
vera_qwen_train.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models
This is the code repo for the paper [`VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models`](https://vera-framework.github.io/) (CVPR 2025).

Our paper presents a video anomaly detection (VAD) method utilizing vision-language models (VLMs) without the need of using extra reasoning modules or instruction tuning datasets: **we just need to input the prompts found by verbalized learning with a given VLM to get VAD results.** This style will eliminate the construction of fine-tuning dataset or the introduction of additional LLMs like ChatGPT.

**What can this repo do for us?**
With this repo, we can:
* Use the training codes to find the best prompt for your deployed VLMs in video anomaly detection (or other vision tasks)
* Use the inference codes and provided prompts/initial scores to run our initial score generation/smoothing codes

**Note**

In this repository, we use InternVL2 as an example to illustrate the use of the provided open-source code.

## 0. Environment and Data

The construction of environment for running VERA relies on the backbone we choose. Please install the environment based on the backbone we want to run VERA on. For example, in this repo, we use InternVL2, we will install the environment based on the instruction provided by InternVL2 (https://internvl.readthedocs.io/en/latest/get_started/installation.html). We also install the PyTorch Lighting (https://lightning.ai/docs/pytorch/stable/) package.

For datasets, we run on UCF-Crime and XD-Violence. Please download the original videos from links provided by the authors. UCF-Crime's project page is https://www.crcv.ucf.edu/projects/real-world/ (we recommend using the dropbox link) and XD-Violence's project page is https://roc-ng.github.io/XD-Violence/. Please let us know if you cannot download from the official links and we are happy to help.

With the downloaded videos, we need to extract frames from the original videos. Please use the provided code in the `Preprocessing` folder for getting extracted frames. After that, we can put the frames in the `Data` folder

## 1. Training in VERA
We can run the following script to obtain the optimal guiding question. The default batch size is 2 and the number of sampled frames is 8. After training, we usually choose the one with the highest validation accuracy as the guiding questions used in testing (we can also use AUC as the metric for selecting the guiding questions).
```
python training.py
```
**Note: If you would like to train on XD-Violence, please uncomment L303-L305 in training.py. This will load the XD-Violence training and validation data.**

For UCF-Crime, the best guiding questions we find are as follows:
```
1. Are there any people in the video who are not in their typical positions or engaging in activities that are not consistent with their usual behavior?
2. Are there any vehicles in the video that are not in their typical positions or being used in a way that is not consistent with their usual function?
3. Are there any objects in the video that are not in their typical positions or being used in a way that is not consistent with their usual function?
4. Is there any visible damage or unusual movement in the video that indicates an anomaly?
5. Are there any unusual sounds or noises in the video that suggest an anomaly?
```

For XD-Violence, the best guiding questions we find are as follows:
```
1. Are there any frames where the video content shows an object or person performing an unusual action that does not fit the usual pattern of the video content?
2. Are there any frames where the video content shows an object or person behaving in a way that is clearly abnormal or unexpected?
3. Are there any frames where the video content shows a clear indication of an anomaly in the behavior of objects or people, such as an object moving in an unexpected way or a person performing an unusual action?
4. Are there any frames where the video content shows a player or object being out of place or not fitting the usual pattern of the video content?
5. Are there any frames where the video content shows a disruption in the normal flow of the video, such as a sudden change in the camera angle or a loss of video quality?
```

## 2. Inference in VERA 
After finishing training and deciding the guiding questions we will use, we need to do inference for test videos. The related codes are put in the `Inference` folder. We can put the learned questions in the backbone and run them on test data to get the initial anomaly score in the segment level. This is Step 1 in VERA's testing (Please refer to Sec. 3.3. in our paper). For this step, we need to run this script after putting guiding questions in the variable ```new_qs``` in the script.

```
python generate_initial_score.py
```

**Note**: We have provided the initial scores obtained by VERA (backbone: InternVL2) on UCF-Crime with this [link](https://drive.google.com/file/d/1LIvPUE7Pv7tPxtXjpOU0sOpSZs6Y7xGv/view?usp=drive_link). We have also provided the initial scores obtained by VERA (backbone: InternVL2) on XD-Violence with this [link](https://drive.google.com/file/d/113rhslKnIE7CKeWd0HKiy8FWp0QJ5OxH/view?usp=drive_link).

Next, given the initial score, we need to run Step 2 and Step 3 introduced in Sec. 3.3 for getting the frame-level anomaly score. We put those two steps in one script and we just need to run this line:

```
python compute_auc.py
```

Please note that the inference scores requires us to extract the vision feature beforehand. We use LAVAD's (https://github.com/lucazanella/lavad) and provide code and use ImageBind for extracting the vision features of each segment. You can access the extracted features we use for UCF-Crime from this [link](https://drive.google.com/file/d/1gwkgIKiPg6fpRaO5yXkxuRcN3X1-97LF/view?usp=drive_link) directly. The extracted feature for XD-Violence is provide in this [link](https://drive.google.com/file/d/1_bN5TRYPpaFpLpf_M7YniRwWeOZgNJtI/view?usp=sharing).

**Note**: We have provide the computation of AUC and AP for XD-Violence dataset as well, which is in the code ```compute_auc_XD_Violence.py``` and ```compute_AP_XD_Violence.py```, respectively. For AP computation in XD-Violence, we follow [Wu et al.](https://roc-ng.github.io/XD-Violence/) for the preprocessing and postprocessing of getting ground truth and smoothing.

## Acknolwedgement
We thank [Zanella et al.](https://github.com/lucazanella/lavad), [Lv et al.](https://github.com/ktr-hubrt/VAD-LLaMA), and [Wu et al.](https://roc-ng.github.io/XD-Violence/) for sharing their codes.

## Citation
If you use this code or find our work helpful, please consider citing our paper. Thank you!
```
@InProceedings{Ye_2025_CVPR,
    author    = {Ye, Muchao and Liu, Weiyang and He, Pan},
    title     = {VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {8679-8688}
}
```