captions-in-VQA

KO | EN

gitlite — search

#hacktoberfest #react #ai #llm #golang #android #machine-learning #docker #macos #nodejs #awesome-list #security #mcp #openai

captions-in-VQA

(★ 19)

Using image captions with LLM for zero-shot VQA

captions-in-VQA 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
Work  in Progress.

## Introduction
This is the official code for the paper "Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts". The code evaluates the effect of using image captions with LLMs for zero-shot Visual Question Answering (VQA).

### Directory Structure

- **/predictions**: This directory contains the predictions generated by GPT-3.5 QA, CogVLM and BLIP-2 VQA models.
- **/captions**: This directory contains the image caption generated by several approaches.
```
cd captions-in-VQA
pip install -r requirements.txt
```

### Create image captions
The following script creates question-driven captions using KeyBERT(https://github.com/MaartenGr/KeyBERT) and CogVLM (https://github.com/THUDM/CogVLM).
You should install the requirements for these two packages before running the script.
The script is a modified version of https://github.com/THUDM/CogVLM/blob/main/basic_demo/cli_demo_hf.py 
```
python write_qd_captions.py --from_pretrained THUDM/cogvlm-chat-hf --bf16 --quant 4 --image-path path-to-images --question-path path-to-questions
```

### Generate answers through GPT-3.5 using image captions
Modify the OPENAI_API_KEY in the script before running.
```
# generate answers using question-driven captions
python qa.py --caption-file captions/cogvlm_chat_qd_descriptions.json --question-file path-to-questions
```

### Evaluate VQA predictions
```
# evaluate question-driven image captioning in VQA
python evaluation.py --prediction-file predictions/cogvlmqd_cap_gpt35.json --question-file path-to-questions
```

## Zero-shot Evaluation Results
We evaluate several VQA approaches on GQA testdev-balanced accross various question types.

| Question types | CogVLM-C Cap. + GPT-3.5 | CogVLM-V Cap. + GPT-3.5 | CogVLM QD Cap. + GPT-3.5 | CogVLM SB Cap. + GPT-3.5 | FuseCap + GPT-3.5 | BLIP-2 Cap. + GPT-3.5 |
|----------------|-------------------------|-------------------------|--------------------------|--------------------------|-------------------|-----------------------|
| verify         | 63.01                   | 58.53                   | 66.83                    | 61.06                    | 53.60             | 55.82                 |
| query          | 36.91                   | 31.08                   | 38.34                    | 31.51                    | 29.61             | 31.87                 |
| choose         | 65.25                   | 60.90                   | 65.51                    | 60.90                    | 58.07             | 60.82                 |
| logical        | 59.51                   | 60.29                   | 59.07                    | 58.46                    | 57.07             | 56.07                 |
| compare        | 51.78                   | 51.95                   | 51.95                    | 49.07                    | 54.50             | 48.22                 |
| object         | 61.95                   | 63.24                   | 59.13                    | 58.87                    | 59.38             | 58.35                 |
| attribute      | 51.75                   | 46.42                   | 54.62                    | 50.80                    | 45.11             | 46.63                 |
| category       | 47.35                   | 44.21                   | 50.39                    | 42.56                    | 43.52             | 42.47                 |
| relation       | 42.56                   | 38.32                   | 42.97                    | 35.76                    | 34.98             | 37.23                 |
| global         | 49.04                   | 45.86                   | 45.86                    | 44.59                    | 43.95             | 45.22                 | 
| TOTAL          | 48.06                   | 43.83                   | 49.50                    | 44.12                    | 41.58             | 42.99                 |

captions-in-VQA

captions-in-VQA 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

captions-in-VQA

captions-in-VQA 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README