R2P

KO | EN

gitlite — search

#javascript #typescript #llm #go #machine-learning #rust #deep-learning #linux #docker #macos #cli #kubernetes #vue #mcp

(★ 25)

Official codebase for the paper "Training-Free Personalization via Retrieval and Reasoning on Fingerprints"

R2P Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Training-Free Personalization via Retrieval and Reasoning on Fingerprints

[![ICCV 2025](https://img.shields.io/badge/ICCV-2025-blue.svg)](https://iccv.thecvf.com/)
[![arXiv](https://img.shields.io/badge/arXiv-2503.18623-b31b1b.svg)](https://arxiv.org/abs/2503.18623)
[![Project Page](https://img.shields.io/badge/Project-Page-green.svg)](https://deepayan137.github.io/papers/training-free-personalization.html)

> **Training-Free Personalization via Retrieval and Reasoning on Fingerprints**  
> Deepayan Das, Davide Talon, Yiming Wang, Massimiliano Mancini, Elisa Ricci
> *International Conference on Computer Vision (ICCV) 2025*

## 📋 Abstract

This repository contains the official implementation of **R2P (Retrieval and Reasoning for Personalization)**, a novel training-free framework that enables personalization using only pre-trained Vision-Language Models (VLMs). Our approach demonstrates for the first time that training-free personalization is feasible by leveraging retrieval and reasoning mechanisms on fingerprints.

## 🚀 Key Features

- **Training-Free**: No fine-tuning required - works with pre-trained VLMs
- **Novel R2P Framework**: Combines retrieval and reasoning for personalization
- **Fingerprint-Based**: Uses fingerprints as the basis for personal concept inference
- **Efficient**: Minimal computational overhead compared to training-based methods

## 📁 Repository Structure

```
├── LICENSE
├── README.md
├── requirements.txt       # Python dependencies
├── data/                  # Raw datasets (YoLLaVA, MyVLM, PerVA)
├── eval_files/           # Generated evaluation files
├── example_database/     # Example database for quick testing
├── scripts/              # Utility scripts
└── src/                  # Source code
    ├── main.py           # Main evaluation script
    ├── detector.py       # Detection components
    ├── retriever.py      # Retrieval system
    ├── database/         # Database creation and management
    │   ├── create_db.py
    │   ├── create_train_test_split.py
    │   ├── create_train_test_perva_split.py
    │   ├── create_ret_files.py
    │   └── mini_cpm_info.py
    ├── models/           # Model implementations
    │   ├── model_interface.py
    │   ├── model_adapters.py
    │   ├── prompt_generator.py
    │   ├── mini_cpm_reasoning.py
    │   ├── internvl_reasoning.py
    │   └── qwen_reasoning.py
    ├── evaluators/       # Evaluation metrics
    │   ├── compute_accuracy.py
    │   └── compute_confidence.py
    └── utils/            # Utility functions
        ├── defined.py
        ├── helpers.py
        └── generate_report.py
```

## 🛠️ Installation

### Prerequisites
- Python >= 3.8
- CUDA >= 11.0 (for GPU support)
- 16GB+ RAM recommended

### Environment Setup

1. **Clone the repository**
   ```bash
   git clone https://github.com/Deepayan137/R2P
   cd R2P
   ```

2. **Create virtual environment**
   ```bash
   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   ```

3. **Install dependencies**
   ```bash
   pip install -r requirements.txt
   ```

### Verify Installation
```bash
python -c "import torch; print(f'PyTorch version: {torch.__version__}')"
python -c "import transformers; print('Transformers installed successfully')"
```

## 📊 Dataset Setup

### Download Datasets
Our experiments use three datasets for personalization evaluation:

1. **YoLLaVA Dataset**
2. **MyVLM Dataset** 
3. **PerVA Dataset**

For YoLLaVA and MyVLM, Please download the data used in the paper from the repositories of [MyVLM](https://snap-research.github.io/MyVLM/) and [Yo'LLaVA](https://thaoshibe.github.io/YoLLaVA/).

For PerVA download the dataset from [here](https://drive.google.com/file/d/1VntjTpEIW40hLMhvoIqjATzs1Tjiry31/view?usp=sharing)

### Dataset Structure
After downloading, your data directory should contain:
```
data/
├── data-yollava/              # YoLLaVA dataset
├── data-myvlm/                # MyVLM dataset
└── data-perva/                # PerVA dataset
```

## 🚀 Setup and Preparation

### Step 1: Create Train-Test Splits

Before running the main evaluation, you need to generate train-test splits for each dataset:

**For YoLLaVA and MyVLM datasets:**
```bash
# Generate train-test split for MyVLM
python src/database/create_train_test_split.py --dataset MyVLM --seed 23

# Generate train-test split for YoLLaVA  
python src/database/create_train_test_split.py --dataset YoLLaVA --seed 23
```

**For PerVA dataset:**
```bash
# PerVA uses a different split creation script
python src/database/create_train_test_perva_split.py --seed 23
```

### Step 2: Create Database for Retrieval

Create the retrieval database for each dataset:

```bash
# Create database for MyVLM
python src/database/create_db.py --dataset MyVLM --seed 23 --user_defined

# Create database for YoLLaVA
python src/database/create_db.py --dataset YoLLaVA --seed 23 --user_defined

# Create database for PerVA
python src/database/create_db.py --dataset PerVA --seed 23 --user_defined
```

### Step 3: Generate Evaluation Files

Create the evaluation files needed for testing:

```bash
# Generate evaluation files for YoLLaVA
python ssrc/database/create_eval_files.py --dataset YoLLaVA  --seed 23

# Generate evaluation files for MyVLM
python ssrc/database/create_eval_files.py --dataset MyVLM  --seed 23

# Generate evaluation files for PerVA
python ssrc/database/create_eval_files.py --dataset PerVA  --seed 23
```
## 🏃‍♂️ Running Experiments

### Main Evaluation Command

After completing the setup steps, run the main evaluation with the following command structure:

```bash
python src/main.py --concept_name $CONCEPT_NAME --category_name $CATEGORY_NAME \
--seed $SEED --data_name $DATA_NAME --rerank_early --refined_k 3 \
--attribute_based_step_by_step --two_step --input_type text --only_recall --model_name $MODEL_NAME
```

### Dataset-Specific Usage

**For YoLLaVA and MyVLM datasets:**
```bash
# Example: Run on MyVLM dataset
python src/main.py --concept_name "person_john" --category_name "all" \
--seed 23 --data_name "MyVLM" --rerank_early --refined_k 3 \
--attribute_based_step_by_step --two_step --input_type text --only_recall --model_name mini_cpm

# Example: Run on YoLLaVA dataset  
python src/main.py --concept_name "dog_max" --category_name "all" \
--seed 23 --data_name "YoLLaVA" --rerank_early --refined_k 3 \
--attribute_based_step_by_step --two_step --input_type text --only_recall --model_name mini_cpm
```

**For PerVA dataset:**
```bash
# Example: Run on PerVA dataset with specific category
python src/main.py --concept_name "my_bag" --category_name "bag" \
--seed 23 --data_name "PerVA" --rerank_early --refined_k 3 \
--attribute_based_step_by_step --two_step --input_type text --only_recall --model_name mini_cpm
```

### Available Categories

**YoLLaVA and MyVLM**: Use `"all"` as category_name

**PerVA**: Choose from the following categories:
- `bag` `book` `bottle` `bowl` `clothe` `cup` `decoration` `headphone` `pillow` `plant` `plate` `remote` `retail` `telephone` `tie` `towel` `toy` `tro_bag` `tumbler` `umbrella` `veg`

### Available Models

The framework supports multiple VLM models:
- **mini_cpm**: Mini-CPM model (default)
- **internvl**: InternVL model  
- **qwen**: Qwen model

**Note:** Please follow the instructions for installing dependencies related to qwen and intervl from their respective HuggingFace repositories. The present codebase has not been extensively tested with internvl and qwen.

### Key Parameters

- `--concept_name`: Name of the personalized concept (e.g., "person_john", "my_bag")
- `--category_name`: Category/superclass of the concept ("all" for YoLLaVA/MyVLM, specific category for PerVA)
- `--seed`: Random seed for reproducibility (recommended: 23)
- `--data_name`: Dataset name (YoLLaVA, MyVLM, or PerVA)
- `--refined_k`: Number of top retrievals to consider (default: 3)
- `--model_name`: VLM model to use for reasoning
- `--rerank_early`: Enable early reranking
- `--attribute_based_step_by_step`: Use attribute-based reasoning
- `--two_step`: Enable two-step reasoning process
- `--input_type text`: Use text input type
- `--only_recall`: Focus on recall evaluation

### Batch Processing with SLURM

If you have access to a SLURM cluster, you can use the provided job array scripts for efficient batch processing:

```bash
# Submit YoLLaVA evaluation jobs
sbatch scripts/test_yollava.sh 23

# Submit MyVLM evaluation jobs  
sbatch scripts/test_myvlm.sh 23

# Submit PerVA evaluation jobs
sbatch scripts/test_perva.sh 23
```

### Batch Processing without SLURM

If SLURM is not available, you can use bash loop scripts for sequential processing:

```bash
# Make scripts executable
chmod +x scripts/run_yollava_loop.sh
chmod +x scripts/run_myvlm_loop.sh  
chmod +x scripts/run_perva_loop.sh

# Run YoLLaVA evaluation
./scripts/run_yollava_loop.sh 23

# Run MyVLM evaluation  
./scripts/run_myvlm_loop.sh 23

# Run PerVA evaluation
./scripts/run_perva_loop.sh 23
```

**Features of bash loop scripts:**
- Sequential processing of all concepts in a dataset
- Individual log files for each concept (stored in `logs/` directory)
- Progress tracking and error handling
- Same functionality as SLURM scripts but without job scheduling

**Note:** Bash loop scripts process concepts sequentially, which may take longer than SLURM parallel processing.

**Parameters:**
- First argument: seed value (e.g., 23)

### Manual Single-Concept Evaluation

For testing individual concepts or when SLURM is not available, use the manual commands shown above.

## 📊 Results and Report Generation

### Output Structure

After running the `main.py`, results are stored in the following structure:

```
outputs/
└── Mini_CPM_YoLLaVA_seed_23/          # Format: {MODEL_NAME}_{DATASET}_{seed}_{SEED_VALUE}
    ├── concept1.json                   # Individual concept results
    ├── concept2.json
    └── ...
```

Each concept JSON file contains:
- **Recall values**: Caption recall metrics
- **Recognition accuracy**: Classification performance metrics

### Generate Aggregate Reports

To calculate aggregate metrics across all concepts in a dataset:

**For Recall metrics:**
```bash
# YoLLaVA dataset
python src/utils/generate_report.py --model_name mini_cpm --dataset YoLLaVA --seed 23 --name recall_R2P

# MyVLM dataset  
python src/utils/generate_report.py --model_name mini_cpm --dataset MyVLM --seed 23 --name recall_R2P

# PerVA dataset
python src/utils/generate_report.py --model_name mini_cpm --dataset PerVA --seed 23 --name recall_R2P
```

**For Recognition + Recall metrics:**
```bash
# YoLLaVA dataset
python src/utils/generate_report.py --model_name mini_cpm --dataset YoLLaVA --seed 23 --name reco+recall_R2P

# MyVLM dataset
python src/utils/generate_report.py --model_name mini_cpm --dataset MyVLM --seed 23 --name reco+recall_R2P

# PerVA dataset  
python src/utils/generate_report.py --model_name mini_cpm --dataset PerVA --seed 23 --name reco+recall_R2P
```

### Generated Reports

Reports are saved in the following structure:
```
report_23/                                    # Format: report_{SEED}
├── YoLLaVA_mini_cpm_recall_results.json     # Recall metrics
├── YoLLaVA_mini_cpm_reco+recall_results.json # Recognition + Recall metrics
├── MyVLM_mini_cpm_recall_results.json
├── MyVLM_mini_cpm_reco+recall_results.json
├── PerVA_mini_cpm_recall_results.json
└── PerVA_mini_cpm_reco+recall_results.json
```

### Report Parameters

- `--model_name`: Model used for evaluation (mini_cpm, internvl, qwen)
- `--dataset`: Dataset name (YoLLaVA, MyVLM, PerVA)  
- `--seed`: Seed value used in evaluation
- `--name`: Report type (`recall_R2P` or `reco+recall_R2P`)

### Complete Workflow Example

```bash
# 1. Run evaluation for all concepts
sbatch scripts/test_yollava.sh 23
# OR
./scripts/run_yollava_loop.sh 23

# 2. Generate recall report
python src/utils/generate_report.py --model_name mini_cpm --dataset YoLLaVA --seed 23 --name recall_R2P

# 3. Generate recognition + recall report  
python src/utils/generate_report.py --model_name mini_cpm --dataset YoLLaVA --seed 23 --name reco+recall_R2P

# 4. Check results
cat report_23/YoLLaVA_mini_cpm_recall_results.json
```

## 📈 Results

### Main Results

Our R2P framework achieves state-of-the-art performance across three personalization datasets:

#### Recognition Performance (Positive Accuracy)
| Method | MyVLM Dataset | YoLLaVA Dataset | PerVA Dataset |
|--------|---------------|-----------------|---------------|
| MyVLM | 96.6% | 94.9% | 66.0% |
| YoLLaVA | 97.0% | 86.0% | 75.1% |
| RAP | 94.4% | - | 92.9% |
| MiniCPM-o + prompt | 98.5% | 81.2% | 73.0% |
| **R2P (Ours)** | **96.3%** | **91.1%** | **90.2%** |

#### Captioning Performance (Recall)
| Method | MyVLM Dataset | YoLLaVA Dataset | PerVA Dataset |
|--------|---------------|-----------------|---------------|
| MyVLM | 84.7% | 81.6% | 0.3% |
| YoLLaVA | 86.7% | 79.7% | 6.6% |
| RAP | 88.0% | - | 64.1% |
| MiniCPM-o + prompt | 87.4% | 73.9% | 65.7% |
| **R2P (Ours)** | **91.4%** | **87.1%** | **72.5%** |

*Detailed results and analysis can be found in our [paper](https://arxiv.org/abs/2503.18623).*

## ⚖️ License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 📖 Citation

If you find our work useful, please consider citing:

```bibtex
@inproceedings{[your-name]2025r2p,
  title={Training-Free Personalization via Retrieval and Reasoning on Fingerprints},
  author={[Your Name(s)]},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2025}
}
```

## 🙏 Acknowledgments

- The codebase is based on [RAP](https://github.com/Hoar012/RAP-MLLM?tab=readme-ov-file#data). Thanks to the authors.

## 📧 Contact
- **Project Link**: [https://deepayan137.github.io/papers/training-free-personalization.html](https://deepayan137.github.io/papers/training-free-personalization.html)
- **Issues**: Please use the [GitHub Issues](https://github.com/Deepayan137/R2P/issues) for bug reports and feature requests

---

<div align="center">
  <strong>⭐ Star this repo if you find it helpful! ⭐</strong>
</div>

R2P

R2P Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

R2P

R2P Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README