KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #claude-code #codex #dsh #dsh-plugin #ai-agent #ai-agents #deepseek-harness #agent-skills #macos #cli

NEURONS

★ 29 Open GitHub ↗

[ICCV 2025] Neurons: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# **[ICCV 2025]** <span style="font-variant: small-caps;">Neurons</span>: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction

[![arXiv](https://img.shields.io/badge/arXiv-2503.11167-brown?logo=arxiv&style=flat-square)](https://arxiv.org/abs/2503.11167)
[![Hugging Face Model](https://img.shields.io/badge/Hugging%20Face-Model-yellow?logo=huggingface&style=flat-square)](https://huggingface.co/McGregorW/NEURONS)

🌟 **If you find our project useful, please consider giving us a star!** ⭐

</div>

## 📌 Overview

![model](https://raw.githubusercontent.com/xmed-lab/NEURONS/main/assets/framework.png)

*Architecture of the <span style="font-variant: small-caps;">Neurons</span> framework*

</div>

<span style="font-variant: small-caps;">Neurons</span> is a novel framework that emulates the human visual cortex to achieve high-fidelity and interpretable fMRI-to-video reconstruction. Our biologically inspired approach significantly advances the state-of-the-art in brain decoding and visual reconstruction.

## 📣 Latest Updates

🟡 2025/10 &nbsp;&nbsp; Released **model weights, training logs, testing logs, and generated images and videos** — all available at [Hugging Face](https://huggingface.co/McGregorW/NEURONS/tree/main/EXP)! ⚠️ *Note 1:* Cloning the entire `EXP` folder requires **over 60 GB** of storage. To download selectively, use `snapshot_download` with the `allow_patterns` parameter (e.g., `allow_patterns=["EXP/exp_neurons/subj_1/*"]` to download only subject 1). ⚠️ *Note 2:* Due to a server issue, the original weights were lost. We re-cloned the repository and re-ran the experiments. While specific numerical results may vary slightly, the overall performance remains consistent with the paper, which also verifies the reproducibility of this work.

🟡 2025/06 &nbsp;&nbsp; **_Neurons_** is accepted by ICCV-2025!

🟡 2025/04 &nbsp;&nbsp; Code released!

🟡 2025/03 &nbsp;&nbsp; Project launched with paper available on arXiv!

## 🛠️ Installation & Setup

### 🖥️ Environment Setup

We recommend using separate environments for training and testing:

```bash
# Training environment
conda create -n neurons_train python==3.10
conda activate neurons_train
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

# Testing environment (to avoid package conflicts)
conda create -n neurons_test --clone neurons_train
conda activate neurons_test
pip install diffusers==0.11.1
```

### 📊 Data Preparation

1. Download the pre-processed dataset:

```bash
python download_dataset.py
tar -xzvf ./cc2017_dataset/masks/mask_cls_train_qwen_video.tar.gz -C ./cc2017_dataset/masks/
tar -xzvf ./cc2017_dataset/masks/mask_cls_test_qwen_video.tar.gz -C ./cc2017_dataset/masks/
```

2. Run task construction scripts:

```bash
# Rule-based Key Object Discovery
python tasks_construction/find_key_obj.py

# Generate CLIP embeddings
python -m tasks_construction.gen_GT_clip_embeds
```

### ⚙️ Pretrained Weights Preparation

```shell
mkdir pretrained_weights
cd pretrained_weights
wget -O unclip6_epoch0_step110000.ckpt -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/unclip6_epoch0_step110000.ckpt\?download\=true
wget -O last.pth -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/train_logs/final_subj01_pretrained_40sess_24bs/last.pth\?download\=true
wget -O convnext_xlarge_alpha0.75_fullckpt.ckpt -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/convnext_xlarge_alpha0.75_fullckpt.pth\?download\=true
wget -O sd_image_var_autoenc.pth https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/sd_image_var_autoenc.pth\?download\=true
cd ..
```

## 🚀 Quick Start
This codebase allows train, test, and evaluate using one single bash file.

```
bash train_neurons.sh 0 neurons 123456 enhance 1
```

------
Parameters:

`$1`: use which gpu to train

`$2`: train file postfix, e.g, `train_neurons`

`$3`: run which stage: `123456` for the whole process, `3456` for test & eval only

- `1`: train brain model
- `2`: train decoupler
- `3`: recon decoupled outputs, prepare for video reconstruction
- `4`: (Optional) caption the keyframes with BLIP-2 instead of using the outputs of GPT-2 in Neurons
- `5`: video reconstruction
- `6`: evaluation with all metrics

`$4`: inference mode: `['enhance', 'motion']`

`$5`: train which subject: `[0,1,2]`

----
Note that for convenience of debugging, `use_wandb` is set to `False` be default.

If you would like to use wandb, first run `wandb login` and set the `use_wandb` to `True` in `train_neurons.py`.

## 📚 Citation

If you find this project useful, please consider citing:

```bibtex
@inproceedings{wang2025neurons,
  title={Neurons: Emulating the human visual cortex improves fidelity and interpretability in fmri-to-video reconstruction},
  author={Wang, Haonan and Zhang, Qixiang and Wang, Lehan and Huang, Xuanqi and Li, Xiaomeng},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={18367--18376},
  year={2025}
}
```

NEURONS

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README