ZRKGC

KO | EN

gitlite — search

#python #react #ai #typescript #golang #android #rust #docker #nodejs #macos #vue #awesome-list #mcp #open-source

(★ 38)

No description available.

ZRKGC Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Zero-Resource Knowledge Grounded Dialogue Generation

## Dependency

The code has been tested with:

\* python 3.6

\* pytorch 1.1.0

\* Ubuntu 18.04

You first need to create an environment using `anaconda` or `virtualenv` and then activate the env.

Please follow  (https://github.com/microsoft/unilm/tree/master/unilm-v1) to build the unilm environment, especially apex and nlg-eval.

Other dependencies need

```
pip install -r requirements.txt
```
## Data

### Test data

#### Wizard of Wikipedia

Download Wizard of Wikipedia dataset from (https://parl.ai/projects/wizard_of_wikipedia/), then preprocess it by

```
python preprocess/wizard_preprocess.py ${data_path}/test_random_split.json

python preprocess/wizard_preprocess.py ${data_path}/test_topic_split.json
```

#### Topical Chat

Topical Chat dataset can be downloaded from (https://github.com/alexa/alexa-prize-topical-chat-dataset).

Follow the instructions to set  environment and build data. Then preprocess the builded data by

```
python3 preprocess/topical_preprocess.py ${data_path}/alexa-prize-topical-chat-dataset/conversations/test_freq.json ${data_path}/alexa-prize-topical-chat-dataset/reading_sets/post-build/test_freq.json

python3 preprocess/topical_preprocess.py ${data_path}/alexa-prize-topical-chat-dataset/conversations/test_rare.json ${data_path}/alexa-prize-topical-chat-dataset/reading_sets/post-build/test_rare.json
```

#### CMU_DoG

Download CMU_DoG dataset fom (https://github.com/lizekang/ITDD), then preprocess it by

```
python3 preprocess/cmu_dog_preprocess.py ${data_path}/ITDD_data/src-test-tokenized.txt ${data_path}/ITDD_data/tgt-test-tokenized.txt ${data_path}/ITDD_data/knl-test-tokenized.txt
```

The `${data_path}` is location of raw dataset. So you could put the above three raw test datasets under `${data_path}` folder. The processed data will be  placed in the corresponding folder under `test_data` folder .

### Train data

Download and unzip from Google Drive (https://drive.google.com/file/d/1bKjHtJMDwxsXRwQD2UQg37TdEfVg6l3k/view?usp=sharing).  Because of the large size of train data , this process will take some time.

## RUN

### Test:

```
 bash test.sh
```

Our model can be downloaded form from Google Drive (https://drive.google.com/drive/folders/178wJqriC-EQDnX7g3LGbwk0eWDI_CEcM?usp=sharing).

Put it under the  `model` dir and modify the `MODEL_RECOVER_PATH`  in `test.sh`.

To get automatic metrics on 3 test datasets (Wizard, Topical Chat and CMU_DoG),  you should install nlg-eval via (https://github.com/Maluuba/nlg-eval)  or download from Google Drive (https://drive.google.com/drive/folders/1Wejei90e-xPHNNABiPOEuG1JKjeXRwga?usp=sharing) .

This period will spend about 2 hours on a 2X8G 2080Ti machine with the default setting which produces the following results in the paper.

### Train:

Download from (https://github.com/microsoft/unilm/tree/master/s2s-ft) to get `[unilm1.2-base-uncased]` model. The model are trained by using the same model configuration and WordPiece vocabulary as BERT Base.

Note that  `unilm_v2_bert_pretrain` folder shoud  contains  three components: `bert_config.json` 、`unilm1.2-base-uncased.bin`  and  `vocab.txt`. And pls replace 'vocab.txt' with this from Google Drive (https://drive.google.com/file/d/1q-4QBE_H0fulb7_izEABOLbfu1jPGpJZ/view?usp=sharing).

We train ZRKGC on 2 16GB Tesla V100 GPUs in a data parallel manner.

```
bash train.sh
```

Modify the `DATA_DIR`  in `train.sh` to what you download. The default path is `train_data`. Eval every 5000 step during training.  When current F1 is lower than previous F1, the training process will terminate.

This process will spend about 12hours with a batch size 10.

## Citation

If you find ZRKGC useful in your work, you can cite the following paper:
```
@article{li2020zero,
  title={Zero-Resource Knowledge-Grounded Dialogue Generation},
  author={Li, Linxiao and Xu, Can and Wu, Wei and Zhao, Yufan and Zhao, Xueliang and Tao, Chongyang},
  journal={arXiv preprint arXiv:2008.12918},
  year={2020}
}
```

ZRKGC

ZRKGC Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

ZRKGC

ZRKGC Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README