PromptRank

KO | EN

gitlite — search

#javascript #react #typescript #golang #android #machine-learning #deep-learning #linux #nodejs #java #kubernetes #vue #security #mcp

(★ 27)

[ACL 2023] Few-shot Reranking for Multi-hop QA via Language Model Prompting

PromptRank 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Few-shot Reranking for Multi-hop QA via Language Model Prompting

We propose an approach for zero- and few-shot re-ranking of multi-hop document paths for open-domain QA. PromptRank constructs a prompt that consists of **(i) an instruction** and **(ii) the path** and uses an LLM to score paths as probability of generating the question given the prompt.

<img src="img/overview.png" width="700" height="250">

# Instructions

#### 1. Download TF-IDF index
Download the TF-IDF retriever and database for HotpotQA provided by [PathRetriever](https://github.com/AkariAsai/learning_to_retrieve_reasoning_paths) from [this link](https://drive.google.com/open?id=1ra37xtEXSROG_f90XxR4kgElGJWUHQyM) and place its contents in `path-retriever/models`

#### 2. Install requirements 
```
pip install -r requirements.txt
```

#### 3. Downloading processed data
The HotpotQA and 2WikiMQA processed data can be downloaded from [Google Drive] The data is preprocessed by retrieving 200 top TF-IDF articles to seed up inference. (https://drive.google.com/file/d/1mI7XAdHWLhlW6fMOW3LJQMPipSmlnP67/view?usp=share_link). Then unzip the data and place the content it in ```data/```

#### 4.(a) To run PromptRank with a single instruction 
```
python run.py \
--model google/t5-base-lm-adapt \
--eval_batch_size=50 \
--max_prompt_len 600 \
--max_doc_len 230 \
--tfidf_pool_size 100 \
--n_eval_examples 1000 \
--temperature 1.0 \
--eval_data data/hotpotqa/dev.json \
--prompt_template 'Document: Review previous documents and ask some question. Question:'
```

#### 4.(b) To run PromptRank with an ensemble of instructions 
```
python run.py \
--model google/t5-base-lm-adapt \
--eval_batch_size=50 \
--max_prompt_len 600  \
--max_doc_len 230 \
--tfidf_pool_size 100  \
--n_eval_examples 1000 \
--temperature 1.0 \
--eval_data data/hotpotqa/dev.json \
--instruction_template_file instruction-templates/top_instructions.txt
--ensemble_prompts 
```

This uses the top 10 instructions found over HotpotQA which are in `instruction-templates/top_instructions.txt`: 
```python
Document: Review previous documents and ask some question. Question
Document: Review the previous documents and answer question. Question:
Document: Read the previous documents and write the following question. Question:
Document: Search previous documents and ask the question. Question:
To analyze the documents and ask question. Document: Question:
Document: To read the previous documents and write a question. Question:
Document: Read previous documents and write your exam question. Question:
Document: Read the previous documents and ask this question. Question:
Read two documents and answer a question. Document: Question:
Identify all documents and ask question. Document: Question:
```

#### 4.(c) To run PromptRank with a in-context learning (only two demonstrations: one bridge and one yes/no question) 
```
python run.py \
--model google/t5-base-lm-adapt \
--eval_batch_size=50 \
--max_prompt_len 600 \
--max_doc_len 230 \
--tfidf_pool_size 100 \
--n_eval_examples 1000 \
--temperature 1.0 \
--eval_data data/hotpotqa/dev.json \
--prompt_template 'Document: Review previous documents and ask some question. Question:' \
--demos_ids 0,1 \ 
--demos_file data/hotpotqa/in_context_demos.json \
```
#### 4.(d) To run PromptRank with a in-context learning with demonstration ensembling (with 3 ensembles -- each having two demonstrations i.e, 6 demonstrations in total)
```
python run.py \
--model google/t5-base-lm-adapt \
--eval_batch_size=50 \
--max_prompt_len 600 \
--max_doc_len 230 \
--tfidf_pool_size 100 \
--n_eval_examples 1000 \
--temperature 1.0 \
--eval_data data/hotpotqa/dev.json \
--prompt_template 'Document: Review previous documents and ask some question. Question:' \
--demos_ids 0,1 \ 
--demos_file data/hotpotqa/in_context_demos.json \
--n_ensemble_demos 3
```

**Note:** The code supports either instruction ensembling or demonstration ensembling -- not both.

----- 
### Acknowledgements 
We use many components from [PathRetriever](https://github.com/AkariAsai/learning_to_retrieve_reasoning_paths). So thanks to Akari Asai and others for providing their code and models.

#### Citation
If you use this code, please consider citing our paper:
```
@article{promptrank,
  title={Few-shot Reranking for Multi-hop QA via Language Model Prompting},
  author={Khalifa, Muhammad and Logeswaran, Lajanugen and Lee, Moontae and Lee, Honglak and Wang, Lu},
  journal={arXiv preprint arXiv:2205.12650},
  year={2023}
}
```

PromptRank

PromptRank 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

PromptRank

PromptRank 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README