R2-LLM

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 68 Open GitHub ↗

The official GitHub repository of the AAAI-2024 paper "Bootstrapping Large Language Models for Radiology Report Generation".

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
  <h1 align="center">Bootstrapping Large Language Models for Radiology Report Generation</h1>

The official GitHub repository of the AAAI-2024 paper ["Bootstrapping Large Language Models for Radiology Report Generation"](https://ojs.aaai.org/index.php/AAAI/article/view/29826).

# Reference
If our work is helpful to your research, please cite our paper:
``` latex
@inproceedings{chang2024bootstrapping,
  author       = {Chang Liu and
                  Yuanhe Tian and
                  Weidong Chen and
                  Yan Song and
                  Yongdong Zhang},
  editor       = {Michael J. Wooldridge and
                  Jennifer G. Dy and
                  Sriraam Natarajan},
  title        = {Bootstrapping Large Language Models for Radiology Report Generation},
  booktitle    = {AAAI},
  pages        = {18635--18643},
  year         = {2024},
}
```

# Getting Started
1. Before you run the code, you need to create a virtual environment and activate it via the following command:
```bash
conda env create -f environment.yaml
conda activate venv
```

2. Once the virtual environment is created, you need to download the LLM model weights following the instruction in [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4). Once the model weights are downloaded, you need to modify some configuration files:
- `minigpt4/models/minigpt4-7b.yaml`: line 16 with the path of Vicuna 7b model weights.
- `minigpt4/models/minigpt4.yaml`: line 16 with the path of Vicuna 13b model weights.

3. You need to download the dataset from the official websites of [IU X-Ray](https://openi.nlm.nih.gov/faq#collection) and [MIMIC-CXR](https://physionet.org/content/mimic-cxr/2.0.0/). Once the datasets are ready, you need to modify some configuration files:
- `minigpt4/configs/datasets/iuxray/align.yaml`: line 5 with the path of pre-training dataset.
- `minigpt4/configs/datasets/iuxray/generate_then_refine.yaml`: line 5 with the path of IU X-Ray dataset, line 6 with the path of public medical corpora.
- `minigpt4/configs/datasets/mimic/align.yaml`: line 5 with the path of pre-training dataset.
- `minigpt4/configs/datasets/mimic/generate_then_refine.yaml`: line 5 with the path of MIMIC-CXR dataset, line 6 with the path of public medical corpora.

# Training
1. **Pre-training.** We recommend you to follow the instructions below to pre-train MiniGPT-4 on MIMIC-CXR.

(1) Modify the configuration files.
- `train_configs/stage1/config.yaml`: line 12 with the path of the linear projection layer of MiniGPT-4, line 59 with the output path.

(2) Run the following command lines to pre-train MiniGPT-4 on MIMIC-CXR.
```
python train.py --cfg-path train_configs/stage1/config.yaml
```

If you need to reduce the memory usage, we recommend you to use the first stage strategy of `ZeRO` optimizer. Run the following command lines to pre-train MiniGPT-4 on MIMIC-CXR with a lower memory usage.

```
deepspeed --nproc-per-gpu NUM_GPUS --master-port MASTER_PORT train.py --cfg-path train_configs/stage1/config.yaml use_zero_optimizer --deepspeed_config train_configs/stage1/zero.json
```

You can download our pre-trained model weights from [here](https://huggingface.co/a-b-c-d-e-g/R2-LLM).

2. **Fine-tuning.** We recommend you to follow the instructions below to fine-tune MiniGPT-4 on IU X-Ray and MIMIC-CXR.

(1) Modify the configuration files. Herein, we take the IU X-Ray configuration as an example.
- `train_configs/stage2/iuxray/config.yaml`: line 11 with the path of the linear projection layer of pre-trained MiniGPT-4 on MIMIC-CXR, line 56 with the output path.

(2) Run the following command lines to fine-tune MiniGPT-4.

```
python train.py --cfg-path train_configs/stage2/iuxray/config.yaml
```

Our codebase supports `ZeRO` to reduce the memory usage. You can run the following command lines with `ZeRO`.

```
deepspeed --nproc-per-gpu NUM_GPUS --master-port MASTER_PORT train.py --cfg-path train_configs/stage2/iuxray/config.yaml use_zero_optimizer --deepspeed_config train_configs/stage2/iuxray/zero.json
```

You can download our fine-tuned model weights from [here](https://huggingface.co/a-b-c-d-e-g/R2-LLM).

# Inference
Run the following command lines to generate radiology reports.

```
python generate_reports.py \
--cfg-path configs/eval_configs/eval.yaml \
--gpu-id GPU_IDS \
--image_path IMAGE_PATH \
--annotations ANNOTATIONS_PATH_OF_IUXRAY_OR_MIMIC \
--checkpoint PATH_TO_PRETRAINED_MODEL_WEIGHTS \
```

# Acknowledgement
This GitHub repository is heavily built based on the [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4) repository. Thanks to the authors for their great work!

R2-LLM

Today's picks

Was this content helpful?

Recent Feedback

R2-LLM

Today's picks

Was this content helpful?

Recent Feedback

Download README