eraserbenchmark - GitLite

KO | EN

gitlite — search

#javascript #python #ai #typescript #golang #rust #deep-learning #nodejs #macos #windows #awesome #vue #ios #open-source

eraserbenchmark

(★ 102)

A benchmark for understanding and evaluating rationales: http://www.eraserbenchmark.com/

파일 탐색기

최종 버전 다운로드 (.zip)

- boolq.json
- boolq_baas.json
- boolq_bert.json
- boolq_soft.json
- cose_bert.json
- esnli_bert.json
- evidence_inference.json
- evidence_inference_bert.json
- evidence_inference_soft.json
- fever.json
- fever_baas.json
- fever_bert.json
- fever_soft.json
- movies.json
- movies_baas.json
- movies_bert.json
- movies_soft.json
- multirc.json
- multirc_baas.json
- multirc_bert.json
- multirc_soft.json
- - __init__.py
  - encode_attend.py
  - mlp.py
  - model_utils.py
  - sequence_taggers.py
- __init__.py
- metrics.py
- utils.py
.gitignore
data_exploration.ipynb
LICENSE
README.md
REPRODUCTION.txt
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/jayded/eraserbenchmark

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd eraserbenchmark

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

jupyter notebook

브라우저에서 노트북(.ipynb) 파일들을 열람하고 실행할 수 있는 Jupyter 화면을 켭니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# eraserbenchmark
A benchmark for understanding and evaluating rationales: http://www.eraserbenchmark.com/

## Core Files

The core files are [utils](rationale_benchmark/utils.py) and [metrics](rationale_benchmark/metrics.py).
These two files comprise everything you need to work with our released datasets.

[utils](rationale_benchmark/utils.py) documents everything you need to know about our input formats. Output
formats and validation code are covered in [metrics](rationale_benchmark/metrics.py).

## Models

At the moment we offer two forms of pipeline models:
* (Lehman, et al., 2019) - sentence level rationale identification, followed by taking the best resulting sentence and classifying it.
    * Both portions of this pipeline function via encoding the input sentence (via a GRU), attending (conditioned on a query vector), and making a classification.
* BERT-To-BERT - the same as above, but using a BERT model.

### (Lehman, et al., 2019) Pipeline

To run this model, we need to first:
* create a `model_components`, `data`, and `output`, directory
* download GloVe vectors from http://nlp.stanford.edu/data/glove.6B.zip and extract the 200 dimensional vector to `model_components`
* download http://evexdb.org/pmresources/vec-space-models/PubMed-w2v.bin to `model_components`
* set up a virtual env meeting requirements.txt
* download data from the primary website to `data` and extract each dataset to its respective directory
* ensure that we have at least an 11G GPU. Reducing batch sizes may enable running on a smaller GPU.

Then we can run (as an example):
```
PYTHONPATH=./:$PYTHONPATH python rationale_benchmark/models/pipeline/pipeline_train.py --data_dir data/movies --output_dir output/movies --model_params params/movies.json
PYTHONPATH=./:$PYTHONPATH python rationale_benchmark/metrics.py --split test --data_dir data/movies --results output/movies/test_decoded.jsonl --score_file output/movies/test_scores.json
```

### BERT-To-BERT Pipeline

To run this model, instructions are effectively the same as the simple pipeline above, except we also require a GPU with approximately 16G of memory (e.g. Tesla V100). The same caveats about batch sizes apply here as well.

Then we can run (as an example):
```
PYTHONPATH=./:$PYTHONPATH python rationale_benchmark/models/pipeline/bert_pipeline.py --data_dir data/movies --output_dir output_bert/movies --model_params param/movies.json
PYTHONPATH=./:$PYTHONPATH python rationale_benchmark/metrics.py --split test --data_dir data/movies --results output_bert/movies/test_decoded.jsonl --score_file output_bert/movies/test_scores.json
```

For more examples, see the [BERT-to-BERT reproduction](REPRODUCTION.txt).

More models including Lei et al can be found at : https://github.com/successar/Eraser-Benchmark-Baseline-Models