open-eqa - GitLite

KO | EN

gitlite — search

#javascript #python #ai #typescript #golang #machine-learning #docker #java #awesome #kubernetes #vue #mcp #openai #open-source

open-eqa

(★ 365)

OpenEQA Embodied Question Answering in the Era of Foundation Models

파일 탐색기

최종 버전 다운로드 (.zip)

- open-eqa-demo-data.json
- open-eqa-hm3d-demo.mp4
- open-eqa-teaser.mp4
- open-eqa-v0-184-questions.json
- - config.py
  - extract-frames.py
- - extract-frames.py
  - SensorData.py
- frames2videos.py
- open-eqa-v0.json
- README.md
- - __init__.py
  - claude-vision.py
  - gemini-pro-vision.py
  - gemini-pro.py
  - gpt4.py
  - gpt4v.py
  - llama.py
  - README.md
- - __init__.py
  - llm_match.py
- - __init__.py
  - anthropic_utils.py
  - demo_utils.py
  - google_utils.py
  - llama_utils.py
  - openai_utils.py
  - prompt_utils.py
- __init__.py
- blind-llm.txt
- claude3-vision.txt
- gemini-pro-vision.txt
- gpt4v.txt
- mmbench-extra.txt
- mmbench.txt
- README.md
- - index.html
- .gitignore
- app.py
- README.md
.gitignore
CODE_OF_CONDUCT.md
CONTRIBUTING.md
evaluate-predictions.py
LICENSE
open-eqa-quickstart.ipynb
README.md
requirements.txt
setup.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/facebookresearch/open-eqa

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd open-eqa

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

pip install -e .

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# OpenEQA: Embodied Question Answering in the Era of Foundation Models

[[paper](https://open-eqa.github.io/assets/pdfs/paper.pdf)]
[[project](https://open-eqa.github.io)]
[[dataset](data)]
[[bibtex](#citing-openeqa)]

<https://github.com/facebookresearch/open-eqa/assets/10211521/1de3ded4-ff51-4ffe-801d-4abf269e4320>

## Abstract

We present a modern formulation of Embodied Question Answering (EQA) as the task of understanding an environment well enough to answer questions about it in natural language. An agent can achieve such an understanding by either drawing upon episodic memory, exemplified by agents on smart glasses, or by actively exploring the environment, as in the case of mobile robots. We accompany our formulation with OpenEQA – the first open-vocabulary benchmark dataset for EQA supporting both episodic memory and active exploration use cases. OpenEQA contains over 1600 high-quality human generated questions drawn from over 180 real-world environments. In addition to the dataset, we also provide an automatic LLM-powered evaluation protocol that has excellent correlation with human judgement. Using this dataset and evaluation protocol, we evaluate several state-of-the-art foundation models including GPT-4V, and find that they significantly lag behind human-level performance. Consequently, OpenEQA stands out as a straightforward, measurable, and practically relevant benchmark that poses a considerable challenge to current generation of foundation models. We hope this inspires and stimulates future research at the intersection of Embodied AI, conversational agents, and world models.

## Dataset

The OpenEQA dataset consists of 1600+ question answer pairs $(Q,A^*)$ and corresponding episode histories $H$.

The question-answer pairs are available in [data/open-eqa-v0.json](data/open-eqa-v0.json) and the episode histories can be downloaded by following the instructions [here](data).

**Preview:** A simple tool to view samples in the dataset is provided [here](viewer).

## Baselines and Automatic Evaluation

### Installation

The code requires a `python>=3.9` environment. We recommend using conda:

```bash
conda create -n openeqa python=3.9
conda activate openeqa
pip install -r requirements.txt
pip install -e .
```

### Running baselines

Several baselines are implemented in [openeqa/baselines](openeqa/baselines). In general, baselines are run as follows:

```bash
# set an environment variable to your personal API key for the baseline
python openeqa/baselines/<baseline>.py --dry-run  # remove --dry-run to process the full benchmark
```

See [openeqa/baselines/README.md](openeqa/baselines/README.md) for more details.

### Running evaluations

Automatic evaluation is implemented with GPT-4 using the prompts found [here](prompts/mmbench.txt) and [here](prompts/mmbench-extra.txt).

```bash
# set the OPENAI_API_KEY environment variable to your personal API key
python evaluate-predictions.py <path/to/results/file.json> --dry-run  # remove --dry-run to evaluate on the full benchmark
```

## License

OpenEQA is released under the [MIT License](LICENSE).

## Contributors

Arjun Majumdar*, Anurag Ajay*, Xiaohan Zhang*, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, Aravind Rajeswaran

## Citing OpenEQA

```tex
@inproceedings{majumdar2023openeqa,
  author={Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, Aravind Rajeswaran},
  title={{OpenEQA: Embodied Question Answering in the Era of Foundation Models}},
  booktitle={{CVPR}},
  year={2024},
}
```