PVIT

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 34 Open GitHub ↗

Official Repository of Personalized Visual Instruct Tuning

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PERSONALIZED VISUAL INSTRUCTION TUNING
Official Repository of the paper: Personalized Visual Instruct Tuning 
<img width="800" alt="image" src="https://github.com/user-attachments/assets/91a21871-cd8c-4beb-b9d5-ce16a6c1cf1a">

# Update
- Release the training script ```personalize-llava/scripts/personalization/finetune_task_lora.sh```.
- PVIT-3M dataset has released in [Huggingface](https://huggingface.co/datasets/Sterzhang/PVIT-3M).
- Our paper is now available at: [https://arxiv.org/abs/2410.07113](https://arxiv.org/abs/2410.07113).

# Introduction
Recent advancements in multimodal large language models (MLLMs) have demonstrated significant progress; however, these models exhibit a notable limitation,
which we refer to as “face blindness”. Specifically, they can engage in general
conversations but fail to conduct personalized dialogues targeting at specific individuals. This deficiency hinders the application of MLLMs in personalized settings,
such as tailored visual assistants on mobile devices, or domestic robots that need
to recognize members of the family. In this paper, we introduce Personalized
Visual Instruction Tuning (PVIT), a novel data curation and training framework
designed to enable MLLMs to identify target individuals within an image and
engage in personalized and coherent dialogues. Our approach involves the development of a sophisticated pipeline that autonomously generates training data
containing personalized conversations. This pipeline leverages the capabilities of
various visual experts, image generation models, and (multi-modal) large language
models. To evaluate the personalized potential of MLLMs, we present a benchmark
called P-Bench, which encompasses various question types with different levels of
difficulty. The experiments demonstrate a substantial personalized performance
enhancement after fine-tuning with our curated dataset.

# Case Study
<img width="1000" alt="image" src="https://github.com/user-attachments/assets/d50fa03f-fdb6-41ff-ab25-806578d29f3e">

# Dataset Statistic
<img width="900" alt="image" src="https://github.com/user-attachments/assets/a208273e-e74b-43a2-8f90-7933a4d026e6">

# Citation
If you find our work useful, please cite using this BibTeX:

```bibtex
@misc{pi2024personalizedvisualinstructiontuning,
      title={Personalized Visual Instruction Tuning}, 
      author={Renjie Pi and Jianshu Zhang and Tianyang Han and Jipeng Zhang and Rui Pan and Tong Zhang},
      year={2024},
      eprint={2410.07113},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.07113}, 
}
```

PVIT

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README