EchoSight

KO | EN

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #dsh #claude-code #codex #developer-tools #react

★ 91 Open GitHub ↗

[EMNLP 2024 Findings] The official PyTorch implementation of EchoSight: Advancing Visual-Language Models with Wiki Knowledge.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# EchoSight: Advancing Visual-Language Models with Wiki Knowledge (EMNLP 2024 Findings)
This is the official PyTorch implementation of EchoSight: Advancing Visual-Language Models with Wiki Knowledge.

[[Project Page]](https://go2heart.github.io/echosight) [[Paper]](https://arxiv.org/abs/2407.12735)

## Requirements
1. (Optional) Create conda environment

```bash
conda create -n echosight python=3.10
conda activate echosight
```

2. Install the required packages
```bash
pip install -r requirements.txt
```
## Knowledge Base
We provide the knowledge bases used in EchoSight. The knowledge base file is the same format as the Encyclopedic-VQA dataset. Apart from the original 2M knowledge base for Encyclopedic-VQA, we also provide a 100K knowledge base for InfoSeek, which is a filtered subset of the 2M knowledge base. The knowledge base files can be downloaded from the following links:
### Enclyclopedic-VQA
- [Encylopedic-VQA's 2M Knowledge Base](https://storage.googleapis.com/encyclopedic-vqa/encyclopedic_kb_wiki.zip)
- [Enclopedic-VQA KB Images Faiss Index](https://www.dropbox.com/scl/fi/hphfse044hfvlnexs4k6s/evqa_2M_faiss_index.zip?rlkey=glrb8lcpd62rqpca2l2tawd4f&st=f8kuwlhy&dl=0)
### Infoseek
- [Our InfoSeek's 100K Knowledge Base ](https://www.dropbox.com/scl/fi/dz37pfa3p6ikv9bcrwv12/infoseek_100k_wiki.zip?rlkey=vnffwca4ojlq4v3fvdxmzlaih&st=yrzi7c1b&dl=0)
- [InfoSeek KB Images Faiss Index](https://www.dropbox.com/scl/fi/dagf0ss39697kq4w0rsli/infoseek_100k_faiss_index.zip?rlkey=cm069zdhnss4xqx5tgy8iaz23&st=fm603c8o&dl=0)

## VQA Questions
### Encyclopedic VQA
The VQA questions can be downloaded in .csv format here(the val and test set is provided by Encyclopedic-VQA, while the training is our cleaned version due to some missings in the repackaged image pixels):

*   [train.csv](https://www.dropbox.com/scl/fi/a954bgai9fafvqpfl6ik8/train_full_image_cleaned.csv?rlkey=wpwmnbkibbt48zygmhdri3h77&st=72ixocy2&dl=0)
*   [val.csv](https://storage.googleapis.com/encyclopedic-vqa/val.csv)
*   [test.csv](https://storage.googleapis.com/encyclopedic-vqa/test.csv)

To download the images in Encyclopedic-VQA:

- [iNaturalist 2021](https://github.com/visipedia/inat_comp/tree/master/2021)(Also put id2name file in the same folder, which can be downloaded from [train_id2name](https://www.dropbox.com/scl/fi/n3jqliyfcpsvn5nso422e/train_id2name.json?rlkey=c0xzyejqacc8hebbci3b8pj3h&st=v5yo2g93&dl=0) and [val_id2name](https://www.dropbox.com/scl/fi/stqfcltlei3j3qlqh7hv3/val_id2name.json?rlkey=a1e1xn2uv70rig3p8u9pf4vdc&st=1l7leqso&dl=0)

- [Google Landmarks Dataset V2](https://github.com/cvdfoundation/google-landmark)

### InfoSeek
The VQA questions of InfoSeek are transformed to E-VQA format from the original InfoSeek dataset. The questions can be downloaded in .csv format here:
* [train.csv](https://www.dropbox.com/scl/fi/lh9b6u62eh391d1fn74pn/infoseek_train_filtered.csv?rlkey=lgfh7fr0ycsgw9x6kk5ym086n&st=8cw6styg&dl=0)
* [test.csv](https://www.dropbox.com/scl/fi/o2drl5usos02370jptnn0/infoseek_test_filtered.csv?rlkey=459z1tcq2viw2r0c1l12s54uw&st=2m8ujpte&dl=0)

To download the images in InfoSeek:

- [Oven](https://github.com/edchengg/oven_eval/tree/main/image_downloads)

## Training
The multimodal reranker of EchoSight is trained using Encyclopedic-VQA datasets and the corresponding 2M Knowledge Base. If you want to enable Hard Negative Sampling when training the reranker, we provide our Hard_Neg result sampled by Eva-CLIP here:
- [Hard Negative Sampling File](https://www.dropbox.com/scl/fi/ft9xkmo4t2p5vmkrk8uko/E_VQA_Hard_Neg.json?rlkey=brzvzcrd7tsxkm1t778d0nwd6&st=ljz3tu3r&dl=0)

To train the multimodal reranker, run the bash script after changing the necessary configurations.
```bash
bash scripts/train_reranker.sh
```
### Script Details
The train_reranker.sh script is used to fine-tune the reranker module with specific parameters:

--`blip-model-name`: Name of the BLIP model to be used for reranking.

--`num-epochs`: Number of epochs for training. In this case, the model will be trained for 20 epochs.

--`num-workers`: Number of worker threads for data loading.

--`learning-rate`: Learning rate for the optimizer.

--`batch-size`: Number of samples per batch during training.

--`transform`: Transformation applied to the data. targetpad ensures the data is padded to a target size.

--`target-ratio`: Target aspect ratio for the padding transformation.

--`save_frequency`: Frequency (in steps) to save the model checkpoints.

--`train_file`: Path to the training data file. The training file should be the same format as provided by Encyclopedic-VQA.

--`knowledge_base_file`: Path to the knowledge base file in JSON format. The format should be the same with that of the Encyclopedic-VQA.

--`negative_db_file`: Path to the hard negative sampled database file used for training.

--`inat_id2name`: Path to the iNaturalist ID to name mapping file.

--`save-training`: Flag to save the training progress.
## Inference
0. Our reranker module weights can be downloaded at [[Checkpoint]](https://www.dropbox.com/scl/fi/qudkqxocz1n2zse9y5mxm/reranker.pth?rlkey=fz2x2r09qetvuxu1zogxf2vkj&st=tvtuql0h&dl=0).

2. To perform inference with the trained model, run the provided test_reranker.sh script after adjusting the necessary parameters.
```bash
bash scripts/test_reranker.sh
```
### Script Details
The test_reranker.sh script uses the following parameters for inference:

--`test_file`: Path to the test file.

--`knowledge_base`: Path to the knowledge base JSON file.

--`faiss_index`: Path to the FAISS index file for efficient similarity search.

--`retriever_vit`: Name of the visual transformer model used for initial retrieval. In the example script, eva-clip is used.

--`top_ks`: Comma-separated list of top-k recall results for retrieval (e.g., 1,5,10,20).

--`retrieval_top_k`: The top-k value used for retrieval.

--`perform_qformer_reranker`: Flag to perform reranking using QFormer.

--`qformer_ckpt_path`: Path to the QFormer checkpoint file.

--`perform_qformer_reranker`: Flag to perform the ultimate VQA.

--`save_result`: Flag to save the inference result.

--`save_result_path`: Path to the result json file would be saved.

--`resume_from`: Path to the retrieval result. If this parameter is used, the inference process will load the saved retrieval result, instead of using the retriever on-the-fly.

2. (Optional) With the saved retrieval or reranked results, an answer generation can be performed standalone.
```bash
bash scripts/test_vqa.sh
```
3. (Optional) Run the batch inference vqa script (Releasing Soon).
### Script Details
The test_vqa.sh script uses the following parameters for inference:

--`test_file`: Path to the test file.

--`retrieval_results`: Path to the retrieval result file.

--`answer_generator`: Name of the answer generator model to be used. Choose from [Mistral, LLaMA3, GPT4, PaLM].

--`llm_checkpoint`: Path to the [Mistral](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) or [LLaMA3](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) checkpoint file. If using GPT4 or PaLM, this parameter is not needed. Instead, change api_key in model/anwser_generator.py.

--`output_file`: Path to the output file. Default is ./answer.json.
## Demo
Run the demo of EchoSight.
```bash
python app.py
```
### Demo Showcase
<img width="1267" alt="image" src="https://github.com/Go2Heart/EchoSight/assets/71871209/7daa8ee7-5b3d-4789-ba43-16c481471b77">

## Citation
```
@inproceedings{yan-xie-2024-echosight,
    title = "{E}cho{S}ight: Advancing Visual-Language Models with {W}iki Knowledge",
    author = "Yan, Yibin  and
      Xie, Weidi",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.83",
    pages = "1538--1551",
    abstract = "Knowledge-based Visual Question Answering (KVQA) tasks require answering questions about images using extensive background knowledge. Despite significant advancements, generative models often struggle with these tasks due to the limited integration of external knowledge. In this paper, we introduce **EchoSight**, a novel multimodal Retrieval-Augmented Generation (RAG) framework that enables large language models (LLMs) to answer visual questions requiring fine-grained encyclopedic knowledge. To strive for high-performing retrieval, EchoSight first searches wiki articles by using visual-only information, subsequently, these candidate articles are further reranked according to their relevance to the combined text-image query. This approach significantly improves the integration of multimodal knowledge, leading to enhanced retrieval outcomes and more accurate VQA responses. Our experimental results on the E-VQA and InfoSeek datasets demonstrate that EchoSight establishes new state-of-the-art results in knowledge-based VQA, achieving an accuracy of 41.8{\%} on E-VQA and 31.3{\%} on InfoSeek.",
}
```
## Acknowledgements
Thanks to the code of [LAVIS](https://github.com/salesforce/LAVIS/tree/main) and data of [Encyclopedic-VQA](https://github.com/google-research/google-research/tree/master/encyclopedic_vqa) and [InfoSeek](https://github.com/open-vision-language/infoseek).

EchoSight

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README