CIPA

KO | EN

gitlite — search

#python #javascript #nodejs #python3 #hacktoberfest #game #go #game-development #deep-learning #pytorch #programming-language #golang

★ 78 Open GitHub ↗

The official code of "Cross-Modal Interactive Perception Network with Mamba for Lung Tumor Segmentation in PET-CT Images", CVPR 2025

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
<h1> Multi-Modal Interactive Perception Network with Mamba for Lung Tumor Segmentation in PET-CT Images </h1>

[Jie Mei](https://jiemei.xyz/)1, Chenyu Lin2, Yu Qiu3, Yaonan Wang1, Hui Zhang1, Ziyang Wang4, Dong Dai4

1 Hunan University,
2 Nankai University,
3 Hunan Normal University,
4 Tianjin Medical University Cancer Institute and Hospital

[![arXiv](https://img.shields.io/badge/arXiv-2503.17261-b31b1b.svg)](https://arxiv.org/abs/2503.17261) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

</div>

- This repository contains the official code for paper [Multi-Modal Interactive Perception Network with Mamba for Lung Tumor Segmentation in PET-CT Images](https://arxiv.org/abs/2503.17261).
- This paper has been accepted to CVPR 2025.
- This code and PCLT20K dataset are licensed for non-commerical research purpose only.

## Introduction

Lung cancer is a leading cause of cancer-related deaths globally. PET-CT is crucial for imaging lung tumors, providing essential metabolic and anatomical information, while it faces challenges such as poor image quality, motion artifacts, and complex tumor morphology. Deep learning-based models are expected to address these problems, however, existing small-scale and private datasets limit significant performance improvements for these methods.
Hence, we introduce a large-scale PET-CT lung tumor segmentation dataset, termed PCLT20K, which comprises 21,930 pairs of PET-CT images from 605 patients.
Furthermore, we propose a cross-modal interactive perception network with Mamba (CIPA) for lung tumor segmentation in PET-CT images. Specifically, we design a channel-wise rectification module (CRM) that implements a channel state space block across multi-modal features to learn correlated representations and helps filter out modality-specific noise. A dynamic cross-modality interaction module (DCIM) is designed to effectively integrate position and context information, which employs PET images to learn regional position information and serves as a bridge to assist in modeling the relationships between local features of CT images. Extensive experiments on a comprehensive benchmark demonstrate the effectiveness of our CIPA compared to the current state-of-the-art segmentation methods. We hope our research can provide more exploration opportunities for medical image segmentation.

![CIPA](https://raw.githubusercontent.com/mj129/CIPA/main/figures/CIPA-method.jpg)

## Environment

1. Create environment.
    ```shell
    conda create -n MIPA python=3.10
    conda activate MIPA
    ```

2. Install all dependencies.
Install pytorch, cuda and cudnn, then install other dependencies via:
    ```shell
    pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
   ```
    ```shell
    pip install -r requirements.txt
    ```

3. Install selective_scan_cuda_core.
    ```shell
    cd models/encoders/selective_scan
    pip install .
    cd ../../..
    ```

## PCLT20K

Please contact Jie Mei (jiemei AT hnu.edu.cn) for the dataset. We will get back to you shortly. The email should contain the following information.
Note: For better academic communication, a real-name system is encouraged and your email suffix must match your affiliation (e.g., hello@hnu.edu.cn). If not, you need to explain why.

```shell
Name: (Tell us who you are.)
Affiliation: (The name/url of your institution or university, etc.)
Job Title: (E.g., Professor, Associate Professor, PhD, etc.)
Email: (Dataset will be sent to this email.)
How to use: (Only for non-commercial use.)
```

## Data Preparation

`train.txt/test.txt` contains the names of items in training/testing set, e.g.:

```shell
 <name1>
 <name2>
 ...
 ```
2. Please put our dataset in the `data` directory

## Usage

### Training
1. Please download the pretrained [VMamba](https://github.com/MzeroMiko/VMamba) weights, and put them under `pretrained/vmamba/`. We use VMamba_Tiny as default.

- [VMamba_Tiny](https://drive.google.com/file/d/1W0EFQHvX4Cl6krsAwzlR-VKqQxfWEdM8/view?usp=drive_link)
    - [VMamba_Small](https://drive.google.com/file/d/1671QXJ-faiNX4cYUlXxf8kCpAjeA4Oah/view?usp=drive_link)
    - [VMamba_Base](https://drive.google.com/file/d/1qdH-CQxyUFLq6hElxCANz19IoS-_Cm1L/view?usp=drive_link)

2. Config setting.

Edit config in the `train.py`.
    Change C.backbone to `sigma_tiny` / `sigma_small` / `sigma_base` to use the three versions of VMamba.

3. Run multi-GPU distributed training:

```shell
    torchrun --nproc_per_node 'GPU_Numbers' train.py
    ```

4. You can also use single-GPU training:

```shell
    python train.py
    ```
5. Results will be saved in `save_model` folder.

### Testing

The pretrained model of CIPA (CIPA.pth) can be downloaded:

- [Baidu Yunpan](https://pan.baidu.com/s/14MfEaSvc-4QFOIWR7w93Tw), Password: CIPA
- [Google Drive](https://drive.google.com/file/d/1x525pjCi4RM51Kv_zbuW7OLx7NPBLRa8/view?usp=sharing)

```shell
python pred.py
```

## Citation

If you are using the code/model provided here in a publication, please consider citing:

```shell
@inproceedings{mei2025cross,
  title={Cross-Modal Interactive Perception Network with Mamba for Lung Tumor Segmentation in PET-CT Images},
  author={Mei, Jie and Lin, Chenyu and Qiu, Yu and Wang, Yaonan and Zhang, Hui and Wang, Ziyang and Dai, Dong},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025}
}
```

## Contact

For any questions, please contact me via e-mail: jiemei AT hnu.edu.cn.

## Acknowledgment

This project is based on the [VMamba](https://github.com/MzeroMiko/VMamba) and [Sigma](https://github.com/zifuwan/Sigma), thanks for their excellent works.

CIPA

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README