DiFusionSeg

KO | EN

gitlite — search

#javascript #react #golang #go #rust #deep-learning #docker #java #nodejs #windows #cli #awesome-list #pytorch #openai

(★ 16)

[KBS 2025] This is official Pytorch implementation of "DiFusionSeg: Diffusion-Driven Semantic Segmentation with Multi-modal Fusion for Perception Optimization"

DiFusionSeg Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DiFusionSeg
This is official Pytorch implementation of "[DiFusionSeg: Diffusion-Driven Semantic Segmentation with Multi-modal Fusion for Perception Optimization](https://www.sciencedirect.com/science/article/pii/S0950705125015205)"
 - 
```
@article{Wang2025DiFusionSegDS,
  title={DiFusionSeg: Diffusion-Driven Semantic Segmentation with Multi-Modal image Fusion for Enhanced Perception},
  author={Zhiwei Wang and Defeng He and Li Zhao and Bo Liu and Yayu Zheng and Xiaoqin Zhang},
  journal={Knowledge-Based Systems},
  year={2025},
  url={https://api.semanticscholar.org/CorpusID:281557716}
}
```
## Framework
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/model_arch.png)

## Recommended Environment
 - [ ] python  3.9.1
 - [ ] torch  1.13.1
 - [ ] cudatoolkit 11.8
 - [ ] torchvision 0.14.0
 - [ ] mmcv  2.2.1
 - [ ] mmcv-full 1.7.2
 - [ ] mmsegmentation 0.30.0
 - [ ] numpy  1.26.4
 - [ ] opencv-python 4.10.0.84

## Experiments 
### Dataset & Checkpoints & Results
The checkpoints and results can be in [DiFusionSeg](https://www.dropbox.com/scl/fo/zjbyp7pml54epiz8wg4gj/AIGFfGfG8Ea_XU25WwyxQno?rlkey=47uzbn6jxn27o4bjvdvo7yk47&st=3gcxs94v&dl=0). Download MSRS dataset from [MSRS](https://pan.baidu.com/s/18q_3IEHKZ48YBy2PzsOtRQ?pwd=MSRS) and the MFNet dataset from [MFNet](https://www.mi.t.u-tokyo.ac.jp/static/projects/mil_multispectral/).
If you need to evaluate other datasets, please organize them as follows:
```
├── /dataset
    MSRS/
    ├── test
    │   ├── ir
    │   ├── Segmentation_labels
    │   ├── Segmentation_visualize
    │   └── vi
    └── train
        ├── ir
        ├── Segmentation_labels
        └── vi
    MFD/
    ├── test
    │   ├── ir
    │   ├── Segmentation_labels
    │   ├── Segmentation_visualize
    │   └── vi
    ├── test_day
    │   ├── ir
    │   ├── Segmentation_labels
    │   └── vi
    ├── test_night
    │   ├── ir
    │   ├── Segmentation_labels
    │   └── vi
    └── train
        ├── ir
        ├── Segmentation_labels
        └── vi
    ......
```
### Evaluate model
python
```
python test_model.py
```
### run sample
python
```
python test_demo.py --img="./images/00131D_vi.png" --ir="./images/00131D_ir.png" --checkpoint="./exps/Done/msrs_vi_ir_meanstd_ConvNext_fusioncomplex_8083/best.pth" --segout="./seg.png"
```
### To Train
Before training DiFusionSeg, you need to download the MSRS dataset MSRS and putting it in ./datasets.

Then running 
python
```
python train_model.py
```
### Segmentation comparison
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/seg.png)
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/MsrsSegTable.png)
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/MfdSegResult.png)
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/MfdSegTable.png)
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/FmbSegResult.png)
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/FmbSegTable.png)
### Fusion comparison
![image](https://raw.githubusercontent.com/warren-wzw/DiFusionSeg/master/images/fusion.png)
## If this work is helpful to you, please cite it as：
```
@article{Wang2025DiFusionSegDS,
  title={DiFusionSeg: Diffusion-Driven Semantic Segmentation with Multi-Modal image Fusion for Enhanced Perception},
  author={Zhiwei Wang and Defeng He and Li Zhao and Bo Liu and Yayu Zheng and Xiaoqin Zhang},
  journal={Knowledge-Based Systems},
  year={2025},
  url={https://api.semanticscholar.org/CorpusID:281557716}
}
```
## Acknowledgements
We use MMSegmentation, DDP, Segformer as the correspond codebase. We would like to express our sincere gratitude to the developers of these codebases.

DiFusionSeg

DiFusionSeg Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

DiFusionSeg

DiFusionSeg Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README