clip2brain

KO | EN

gitlite — search

#llm #python #typescript #machine-learning #ai #nextjs #deep-learning #agent #esp32 #streamlit #ai-agents #nodejs

★ 25 Open GitHub ↗

Code from paper "Natural language supervision with a large and diverse dataset builds better models of human high-level visual cortex"

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# clip2brain
[![DOI](https://zenodo.org/badge/663684836.svg)](https://zenodo.org/badge/latestdoi/663684836)

Code from paper "Natural language supervision with a large and diverse dataset builds better models of human high-level visual cortex"

### Step 1: Clone the code from Github and download the data
```
git clone https://github.com/ariaaay/clip2brain.git
cd clip2brain
```
You will also need to install the [natural scene datasets](https://naturalscenesdataset.org/).
For downstream analysis you might also need to download [coco annotations](https://cocodataset.org/#download). It is optional for just running encoding models. The one used for analysis are: 2017 Train/Val annotations (241MB).

(OPTIONAL) Models in comparison to CLIP (SimCLR, SLIP) can be downloaded at: https://github.com/facebookresearch/SLIP?tab=readme-ov-file, under ViT-base structure.

(OPTIONAL) Pycortex DB for the NSD subjects can be downloaded here: https://drive.google.com/drive/folders/1YXAYggtlsfFxMtiqWs0UrMihxI125MHb?usp=sharing

### Step 2: Install requirements
[Requirements.txt](https://github.com/ariaaay/clip2brain/blob/main/requirements.txt) contains the necessary package for to run the code in this project.
```
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
Install `torch` and `torchvision` from [PyTorch](https://pytorch.org/) according to your own system setup.

### Step 3: Set up paths in `config.cfg`
Modify `config.cfg` to reflect local paths to NSD data and COCO annotations.
See current `config.cfg` for an example.

### Step 4: Reproduce results!
Note: `project_commands.sh` by default runs all models for all subjects. If you would like to speed up the process and only test out certain model, please comment out models you don't need.
```
sh project_commands.sh
```

clip2brain

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README