BrainFLORA

KO | EN

gitlite — search

#python #javascript #machine-learning #typescript #docker #golang #android #docker-compose #flask #natural-language-processing #deep-learning #python3

★ 35 Open GitHub ↗

[ACM MM 2025 Oral] Reproducible multimodal neural embeddings for EEG, MEG, and fMRI visual retrieval, reconstruction, and captioning.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center" style="font-size: 1.8rem;">
  BrainFLORA: Uncovering Brain Concept Representation<br>
  via Multimodal Neural Embeddings
</h1>

<p align="center">
  <a href="https://dongyangli.site/">Dongyang Li</a>,
  <a href="https://github.com/wojiao-yc">Haoyang Qin</a>,
  <a href="https://www.wormforce.net/members/mingyang-wu">Mingyang Wu</a>,
  <a href="https://hedges0-0.github.io/">Chen Wei</a>,
  <a href="https://scholar.google.com/citations?user=UpP9hJ8AAAAJ&hl=en">Quanying Liu</a>
    <br>
  Southern University of Science and Technology
</p>

BrainFLORA is a reproducible multimodal neural embedding framework for EEG, MEG, and fMRI visual retrieval, image reconstruction, and image captioning. The repository provides unified command-line entry points for training, inference, and evaluation with the released checkpoints.

## News

- **2026-06-19**: The codebase was reorganized and Hugging Face checkpoint weights were updated.
- **2025-12-21**: Preprocessed datasets and pretrained checkpoints were released on [Hugging Face](https://huggingface.co/datasets/LidongYang/BrainFLORA).
- **2025-07-15**: The [arXiv paper](https://arxiv.org/abs/2507.09747) was released.
- **2025-07-12**: The codebase was released.
- **2025-07-05**: BrainFLORA was accepted by ACM MM 2025.

## Installation

```bash
git clone https://github.com/ncclab-sustech/BrainFLORA.git
cd BrainFLORA

conda env create -f environment.yml
conda activate BrainFLORA
pip install -e .
```

If you prefer the setup script from the original repository:

```bash
bash setup.sh
conda activate BrainFLORA
pip install -e .
```

Caption evaluation may require optional metric packages such as `clip` and `pycocoevalcap`. Caption generation uses Shikra; download the model assets from the original Shikra project and place them locally:

```text
external_models/shikra-7b
external_models/mm_projector.bin
```

Shikra resources:

- Paper: [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://arxiv.org/abs/2306.15195)
- Code and model instructions: [shikras/shikra](https://github.com/shikras/shikra)
- Hugging Face model page: [shikras/shikra-7b-delta-v1](https://huggingface.co/shikras/shikra-7b-delta-v1)

## Data and Checkpoints

Raw datasets used by the original project:

| Dataset | Link | Dataset | Link |
| --- | --- | --- | --- |
| THINGS-EEG1 | [OpenNeuro ds003825](https://openneuro.org/datasets/ds003825/versions/1.1.0) | THINGS-EEG2 | [OSF 3jk45](https://osf.io/3jk45/) |
| THINGS-MEG | [OpenNeuro ds004212](https://openneuro.org/datasets/ds004212/versions/2.0.0) | THINGS-fMRI | [OpenNeuro ds004192](https://openneuro.org/datasets/ds004192/versions/1.0.7) |
| THINGS-Images | [OSF rdxy2](https://osf.io/rdxy2) | | |

Use `data_preparing/` if you need to preprocess raw data yourself.

## Quick Run Experiments

Run pretrained checkpoint inference and evaluation for visual retrieval, reconstruction, and captioning.

### 1. Visual Retrieval

Evaluate retrieval checkpoints:

```bash
python eval/reproduce_retrieval.py \
  --device cuda:0 \
  --models single unified \
  --modalities eeg meg fmri \
  --batch-size 128 \
  --output-dir outputs/reproduction
```

Outputs:

```text
outputs/reproduction/retrieval_reproduction_metrics.csv
outputs/reproduction/retrieval_reproduction_metrics.json
outputs/reproduction/retrieval_reproduction_vs_paper.json
```

### 2. Visual Reconstruction

Generate reconstructed images with cached SDXL/IP-Adapter assets:

```bash
python eval/FLORA_inference_reconst.py \
  --device cuda:0 \
  --modalities eeg meg fmri \
  --prior-steps 10 \
  --image-steps 4 \
  --local-files-only \
  --skip-existing \
  --output-dir outputs/reconstruction_png_full \
  --summary-json outputs/reconstruction_png_full/reconstruction_png_full_summary.json
```

The full run contains 3100 generated images:

```text
EEG: 10 subjects x 200 images
MEG: 4 subjects x 200 images
fMRI: 3 subjects x 100 images
```

Evaluate reconstruction metrics:

```bash
python eval/evaluate_reconstruction_metrics.py \
  --recon-root outputs/reconstruction_png_full \
  --modalities eeg meg fmri \
  --gt-root /path/to/dataset_root \
  --output-dir outputs/reconstruction_metrics
```

### 3. Visual Captioning

Generate woPrior caption token embeddings and apply the caption diffusion prior:

```bash
python eval/FLORA_inference_caption_embeddings.py \
  --device cuda:0 \
  --stage all \
  --modalities eeg meg fmri \
  --features-root features/FLORA \
  --metrics-json outputs/caption_embedding/caption_woPrior_metrics.json \
  --summary-json outputs/caption_embedding/caption_prior_summary.json
```

Generate caption JSON/TXT files with Shikra:

```bash
python eval/shikra_caption.py \
  --device cuda:0 \
  --modalities eeg meg fmri \
  --conditions Prior woPrior \
  --embeddings-root features/FLORA \
  --output-root Caption \
  --shikra-path external_models/shikra-7b \
  --mm-projector-path external_models/mm_projector.bin \
  --local-files-only
```

Dry-run Shikra caption generation without loading the model:

```bash
python eval/shikra_caption.py \
  --modalities fmri \
  --conditions Prior \
  --subjects sub-01 \
  --embeddings-root features/FLORA \
  --dry-run
```

Evaluate caption metrics:

```bash
python Caption/evaluate_caption_metrics.py \
  --modalities eeg fmri \
  --conditions Prior woPrior \
  --candidate-pattern 'shikra_{tag}_sub_{subject}_caption.json' \
  --output-dir outputs/caption_metrics
```

The default references are:

```text
Caption/EEG_caption/caption_EEG_GT.json
Caption/fMRI_caption/caption_fMRI_GT.json
```

## Quick Training

Unified training scripts are provided for retraining and ablations. Update dataset paths in the configs or pass CLI arguments before launching.

Train retrieval encoders:

```bash
# EEG
python Retrieval/train_retrieval.py \
  --modality eeg \
  --gpu cuda:0 \
  --output-dir outputs/contrast

# MEG
python Retrieval/train_retrieval.py \
  --modality meg \
  --gpu cuda:0 \
  --output-dir outputs/contrast

# fMRI
python Retrieval/train_retrieval.py \
  --modality fmri \
  --gpu cuda:0 \
  --output-dir outputs/contrast
```

Train the unified encoder for retrieval:

```bash
python train/train_unified_encoder.py \
  --task retrieval \
  --modalities eeg meg fmri \
  --gpu cuda:0 \
  --output_dir outputs/contrast
```

Train the unified encoder for reconstruction:

```bash
python train/train_unified_encoder.py \
  --task reconstruction \
  --modalities eeg meg fmri \
  --gpu cuda:0 \
  --output_dir outputs/contrast
```

Train the caption-aligned unified encoder:

```bash
python train/train_unified_encoder.py \
  --task caption \
  --use-caption \
  --modalities eeg meg fmri \
  --gpu cuda:0 \
  --output_dir outputs/contrast
```

Distributed reconstruction training:

```bash
accelerate launch train/train_unified_encoder.py \
  --task reconstruction \
  --distributed \
  --modalities eeg meg fmri \
  --output_dir outputs/contrast
```

## Citation

```bibtex
@inproceedings{li2025brainflora,
  author = {Li, Dongyang and Qin, Haoyang and Wu, Mingyang and Wei, Chen and Liu, Quanying},
  title = {BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings},
  year = {2025},
  isbn = {9798400720352},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3746027.3754996},
  doi = {10.1145/3746027.3754996},
  booktitle = {Proceedings of the 33rd ACM International Conference on Multimedia},
  pages = {5577--5586}
}

@article{li2024visual,
  title = {Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion},
  author = {Li, Dongyang and Wei, Chen and Li, Shiying and Zou, Jiachen and Liu, Quanying},
  journal = {Advances in Neural Information Processing Systems},
  volume = {37},
  pages = {102822--102864},
  year = {2024}
}

@inproceedings{wei2024cocog,
  title = {CoCoG: controllable visual stimuli generation based on human concept representations},
  author = {Wei, Chen and Zou, Jiachen and Heinke, Dietmar and Liu, Quanying},
  booktitle = {Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence},
  pages = {3178--3186},
  year = {2024}
}
```

<h2 style="border-bottom: 1px solid lightgray; margin-bottom: 5px;">😺Acknowledge</h2>

1.Thanks to Y Song et al. for their contribution in data set preprocessing and neural network structure, we refer to their work:"[Decoding Natural Images from EEG for Object Recognition](https://arxiv.org/pdf/2308.13234.pdf)". Yonghao Song, Bingchuan Liu, Xiang Li, Nanlin Shi, Yijun Wang, and Xiaorong Gao.

2.We also thank the authors of [SDRecon](https://github.com/yu-takagi/StableDiffusionReconstruction) for providing the codes and the results. Some parts of the training script are based on [MindEye](https://medarc-ai.github.io/mindeye/) and [MindEye2](https://github.com/MedARC-AI/MindEyeV2). Thanks for the awesome research works.

3.Here we provide the THING-EEG2 dataset cited in the paper: "[A large and rich EEG dataset for modeling human visual object recognition](https://www.sciencedirect.com/science/article/pii/S1053811922008758?via%3Dihub)". Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, Radoslaw M. Cichy.

4.Another used THINGS-MEG and THINGS-fMRI data set provides a reference:"[THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior](https://elifesciences.org/articles/82580.pdf)". Hebart, Martin N., Oliver Contier, Lina Teichmann, Adam H. Rockter, Charles Y. Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I. Baker.

5.We use the "[BrainHub](https://github.com/weihaox/BrainHub)" for visual caption evaluation from "[UMBRAE: Unified Multimodal Brain Decoding (ECCV 2024)](https://dl.acm.org/doi/abs/10.1007/978-3-031-72667-5_14)" Xia, Weihao and de Charette, Raoul and Oztireli, Cengiz and Xue, Jing-Hao.

Contact [Dongyang Li](https://github.com/dongyangli-del) if you have any questions or suggestions.

## License

This repository is released under the MIT license. See [LICENSE](./LICENSE) for details.

BrainFLORA

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README