MReD

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 28 Open GitHub ↗

A Meta-Review Dataset for Controllable Text Generation

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

We provide the full MReD data and its related code.

For more details please refer to the paper: https://aclanthology.org/2022.findings-acl.198/

## 1. Data Information
The `full_data_info.txt` file contains detailed information for the train, validation and test data splits for both classification and summarization tasks.
Each line have 12 fields separated by tab refering to the following information for the same paper:
* **data split**: `val`, `test` or `train`
* **paper id**: `yyyy-id`, where `yyyy` is the year
* **year**: from 2018 to 2021, the year of the ICLR submission 
* **openreview id**: `Paperxxx`, the paper id given by the open review website, can be found in the website's official blind review section, 2nd row in the format of `ICLRyyyy Conference Paperxxx AnonReviewerx`
* **reviewers' rating score**: `Rx: x, Rx: x, ..., Rx: x.` which gives a summary of the score given by each reviewer according to the anonymous reviewer number on openreview website. Note that sometimes the reviewer number is not continuous on openreview (eg., there is only R2, R3, R4)
* **average rating score**: the average score from all reviewers
* **paper decision**: `Reject` or `Accept (Oral)`, `Accept (Poster)`, `Accept (Spotlight)`, `Accept (Workshop)`
* **title**: the title of the paper/submisison
* **link**: the link of the paper/submission on openreview
* **word count**: number of words in meta-review
* **sentence-level category sequence**: `[label1, label2, ... ]`, per-sentence label sequence for the meta-review where `label1` represents the category label for 1st sentence, `label2` for the 2nd sentence and so on
* **segment-level category sequence**: `[label1, label2, ... ]`, label sequence for the meta-review on segment level where `label1` represents the category label for 1st segment (the sentences of the same label), `label2` for the 2nd segment and so on

## 2. Classification
### a. Data
In the folder `classification/data/`, we provide the human-annotated category label for every sentence in all meta-reviews. Each line contains three fields, namely **paper id**, **sentence**, and **label**, separated by tab. Consecutive meta-reviews are separated by an additional new line. You may search the **paper id** in `full_data_info.txt` for more information on a particular paper/submission.

### b. Code
The code for training the classifier is adapted from: https://github.com/allanj/pytorch_neural_crf.
Using Roberta-base, our trained model has a classification accuracy of 85.83%.
We include the our modified code in folder `classification/LSTM-CRF`.
In `classification/LSTM-CRF/data/mred`, we have already adapted our meta-review classification data to the required IOBES tagging format.
To train your own tagger, please follow the original repo for environment setup, then run
`python trainer.py --model_folder <Location to save traind model> --device <CUDA number> --embedder_type roberta-base`.
All hyperparameters are already set in `trainer.py` as default.
The trained model can be located in `classification/LSTM-CRF/model_files/`.
Prediction results are save to `classification/LSTM-CRF/results/`. To obtain the micro and macro F1 scores, run `python eval.py <result_file>`.

## 3. Summarization
### Extractive summarization
#### a. Data
We provide the full data of each paper reviews and metareviews in the source folder `summarization/extractive/data/source` and target folder `summarization/extractive/data/target` respectively, where each line represents the review or the meta-review passage in correspondence. The details of each paper can be found in `full_data_info.txt` following the file order.

The target folder contains the following:
* full target files `{train/val/test}.target`, where each line is a meta-review
* the filtered target files `filtered_{train/val/test}.target` for those with meta-review word count between 20-400, where each line is a meta-review
* the filtered test target low score (<=3) and high score (>=7) files `filtered_test_{low/high}_score.target`, where each line is a meta-review, the meta-review word count is also between 20-400
* the filtered sentence-level labels files `filtered_test_sentence_labels.txt`, `filtered_test_sentence_labels_low_score.txt`, and `filtered_test_sentence_labels_high_score.txt`, where each line is the gold label sequence for the test data split
* the filered per sentence labeled files `filtered_train_labeled_sentences.txt`, `filtered_train_labeled_sentences_low_score.txt`, `filtered_train_labeled_sentences_high_score.txt`, where each line is a labeled sentence and each meta-review is separated by a newline.

The source folder contains the inputs obtained from different linearization methods in folders 
* `concat/`
* `rate-concat/`
* `merge/`
* `rate-merge/`
 Please refer to our paper for details of linearization methods. Inside each source and target folder, we provide a full data version and a filtered data version, where the filtered papers contains meta-review lengths no less than 20 words and no more than 400 words (which can be easily obtained by filtering according to the `full_data_info.txt`'s **word count** field). We used the filtered data in our paper's experiments.

Moreover, we provide the classifier labeled (85.83% accuracy) per-sentence category label for filtered source inputs in `summarization/extractive/data/labeled_source/`, which include:
* the full labeled source files `{train,val,test}_labeled_sentences.txt`
* the filtered paper for meta-review length between 20-400 `filtered_{train,val,test}_labeled_sentences.txt`
* the filtered low score (<=3) and high score (>=7) paper labeled source inputs: `filtered_{train,val,test}_labeled_sentences_low_score.txt` and `filtered_{train,val,test}_labeled_sentences_high_score.txt`.

#### b. Code
We provide code for the three exractive models and the generic baselines in our paper under the directory  `summarization/extractive/code/`.
* **Lexrank**: in folder `lexrank_model/`, based on the package: https://pypi.org/project/lexrank/. We extract the top n most relevant items from src, where n is decided according to the actual number of sentences in gold. The vanilla and labeled-controlled extractive summarization scripts are `lexrank_base.py` and `lexrank_control.py`. Need to run `pip install lexrank`.
* **Textrank**: in folder `textrank_model/`,based on the gensim.summarization.summarizer https://radimrehurek.com/gensim_3.8.3/summarization/summariser.html. We use textrank to extract the top n most relevant items from src, where n is decided according to the actual number of sentences in gold. The vanilla and labeled-controlled extractive summarization scripts are `textrank_base.py` and `textrank_control.py`. Need to run `pip install gensim==3.8.3`.
* **MMR**: in folder `mmr_model/`,based on https://github.com/vishnu45/NLP-Extractive-NEWS-summarization-using-MMR/blob/master/mmr_summarizer.py. The vanilla and labeled-controlled extractive summarization scripts are `mmr_base.py` and `mmr_control.py`. Need to run `pip install gensim==3.8.3`.
* **Generic sentence baselines**: In folder `generic_baseline/`, the script `generic.py` uses textrank to first split all sentences in the input file (src/tgt, high/low papers...) into same label groups, arrange sentences in each group in order and then select according to meta-review label sequence. 
Usage: `python generic.py <input file> <label file> <name>` where name can be `src_generic`, `src_low_score`, `src_high_score`, `tgt_generic`, `tgt_low_score`, `tgt_high_score`.
Examples:
For `Source Low Score`: `python generic.py ../../data/labeled_source/filtered_train_labeled_sentences_low_score.txt ../../data/target/filtered_test_sentence_labels_low_score.txt src_low_score`
For `Target Low Score`: `python generic.py ../../data/target/filtered_train_labeled_sentences_low_score.txt ../../data/target/filtered_test_sentence_labels_low_score.txt tgt_low_score`

For the `Source Generic` baseline (see our paper for details), the input file `../../data/labeled_source/filtered_train_labeled_sentences.txt` may take exceedingly long time to be processed, and a faster method is to process it in chunks, `python generic.py ../../data/labeled_source/filtered_train_labeled_sentences.txt ../../data/target/filtered_test_sentence_labels.txt <chunk id> src_generic` by using chunk id from 0 to 8, then running `python comb_pt.py ../../data/target/filtered_test_sentence_labels.txt 9 src_generic` to combine the processed chunks to produce the output file.

The rouge score evaluation script is included as `eval_rouge.py`. Use `python eval_rouge.py <generated_file> <gold_file>` to run.

The script for linearization to obtain the `merge/` data is also included as `merge.py`.

### Abstractive summarization
#### a. Data
For vanilla generations, we include data in `summarization/absractive/filtered_uncontrolled_data/` with names according to different source linearizaton methods:
* concat: `{train, val, test}_concat.csv`
* rate-concat: `{train, val, test}_rate_concat.csv`
* merge: `{train, val, test}_merge.csv`
* rate-merge: `{train, val, test}_rate_concat.csv`
* longest review: `{train, val, test}_longest.csv`

For label controlled generation, we include data in `summarization/absractive/filtered_controlled_data/` according to different source linearization method similar to vanilla generation, and according to control methods:
* sentence control: file ending with `_sent-ctrl.csv`
* segment control: file ending with `_seg-ctrl.csv`

#### b. Code
We use the example code in Transformers https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py. 
Transformers version: "4.6.0.dev0"
please install datasets with version 1.6.2
Simply download the code from the Transformers and follow the installation instructions. When running the training script, you need to:
* specify the data file path by `--train_file <train_file_path> --validation_file <val_file_path> --test_file <test_file_path>`
* specify the maximum and minimun target generation length (if using filtered files) by `--max_target_length 400 --gen_target_min 20`
* specify the seed using `--seed <number>`
* specify source truncation using `--max_source_length <number>`. Note default transformers only supports up to 1024 tokens.

example usage `CUDA_VISIBLE_DEVICES=0 python run_summarization.py --model_name_or_path facebook/bart-large-cnn --do_train --do_eval --do_predict --train_file filtered_controlled_data/train_rate_concat_sent-ctrl.csv --validation_file filtered_controlled_data/val_rate_concat_sent-ctrl.csv --test_file filtered_controlled_data/test_rate_concat_sent-ctrl.csv --output_dir results/rate_concat_1024_sent-ctrl --overwrite_output_dir --per_device_train_batch_size=1 --per_device_eval_batch_size=1 --predict_with_generate --seed 0 --save_total_limit 1 --max_source_length 1024 --max_target_length 400 --gen_target_min 20`

We have modified the model file to extend the input truncation for BART to more than 1024 tokens (default). You may use our modified file in `summarization/absractive/modeling_utils.py` to replace the file https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py if you wish to read more than 1024 tokens in the source and specify the truncation length by `--max_source_length <length>` when running the training script. The specific section we modified are commented with `# MReD` in `modeling_utils.py`.

## Demo
You can quickly experience the generation quality of our model at https://colab.research.google.com/drive/1VDrP34xTFXtD1VR2Ls-IoC3xCp52OT8L?usp=sharing&pli=1#scrollTo=qUpK8EEpdpp2

## Cite Us
```
@inproceedings{shen2022mred,
  title={MReD: A Meta-Review Dataset for Structure-Controllable Text Generation},
  author={Shen, Chenhui and Cheng, Liying and Zhou, Ran and Bing, Lidong and You, Yang and Si, Luo},
  booktitle={Findings of ACL},
  year={2022}
}
```

MReD

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README