Awesome-Continual-learning-of-Vision-Language-Models

KO | EN

gitlite — search

#python #ai #llm #golang #android #nodejs #windows #cli #kubernetes #awesome-list #vue #pytorch #ios #open-source

Awesome-Continual-learning-of-Vision-Language-Models

(★ 215)

Awsome of VLM-CL. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

파일 탐색기

최종 버전 다운로드 (.zip)

- all_challenges_01.png
- figure2_01.png
- Methods_Overview_01.png
- Metrics_01.png
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Awesome Continual Learning for Vision-Language Models & MLLMs

[![ArXiv](https://img.shields.io/badge/ArXiv-paper-B31B1B.svg?logo=arXiv&logoColor=Red)](https://arxiv.org/pdf/2508.04227v3.pdf)
[![Awesome](https://awesome.re/badge.svg)](https://awesome.re)

> A curated list of papers, codebases, and datasets for continual learning in vision-language models (VLMs) and Multimodal Large Language Models (MLLMs), accompanying our survey paper.
> *Last updated: 2026-08*

---

## 📑 Table of Contents
- [Latest News](#-latest-news)
- [Citing Our Work](#-citing-our-work)
- [Overview & Challenges](#-overview--challenges)
- [Taxonomy](#-taxonomy)
- [🏆 Mini Leaderboard (SOTA Performance)](#-mini-leaderboard-sota-performance)
- [📄 Papers](#-papers)
- [🗂️ Datasets & Benchmarks](#️-datasets--benchmarks)
- [🚀 Future Directions](#-future-directions)

---

## 🔥 Latest News
- **[2026.08]** Our survey paper, **“Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting” (v3)**, is now available on **arXiv**.
- **[2026.03]** We have updated the repository with a new collection of recent papers and resources to reflect the latest advances in **VLM continual learning**.
- **[Ongoing]** This repository is under active maintenance and will be continuously updated with newly released papers, code, and benchmarks from the community.

---

## 📣 Citing Our Work
If you find this repository or our survey useful for your research, please consider citing our work:

```bibtex
@article{liu2026continual,
  title={Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting},
  author={Liu, Yuyang and Hong, Qiuhe and Huang, Linlan and Gomez-Villa, Alexandra and Goswami, Dipam and Liu, Xialei and van de Weijer, Joost and Tian, Yonghong},
  journal={arXiv preprint arXiv:2508.04227},
  year={2026}
}
```

---

## 📜 Overview
Vision-language models (VLMs) like CLIP have achieved remarkable success, but adapting them to evolving data streams without **catastrophic forgetting** is a critical challenge. This repository provides a systematic review of research on continual learning for VLMs (VLM-CL).

Our survey identifies three core failure modes in VLM-CL:
- 🧠 **Cross-Modal Feature Drift**: The degradation of alignment between visual and textual feature spaces during updates.
- ⚙️ **Shared Module Interference**: Destructive weight updates in shared components like cross-attention layers, which is common in fusion-based models.
- 📉 **Zero-Shot Capability Erosion**: The loss of generalization to unseen tasks after the model is fine-tuned on specific downstream tasks.

Based on these challenges, we propose a **solution-driven taxonomy** to categorize existing methods, which forms the structure for the paper list below. This repository serves as a community-driven hub for tracking state-of-the-art methods, datasets, and fostering collaboration.

- 🧠 **Multi-Modal Replay (MMRE)**: *Strategies that mitigate forgetting by revisiting previous knowledge, either through explicit data storage or implicit generation.*
- ⚙️ **Cross-Modal Regularization (CREG)**: *Strategies that preserve alignment by imposing constraints on the model's loss function or update process, often using knowledge distillation or by directly regularizing the feature space.*
- 🛠️ **Parameter-Efficient Adaptation (PEA)**: *Strategies that freeze the main pre-trained model and update only a small number of parameters using prompts, adapters, or low-rank updates to isolate task-specific knowledge.*

---

## 📄 Papers  
*Sorted chronologically (newest first).*

### 2026
* (AAAI) LoRA in LoRA: Towards Parameter-Efficient Architecture
Expansion for Continual Visual Instruction Tuning. [[PDF](https://arxiv.org/pdf/2508.06202)]
* (AAAI) Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning. [[PDF](https://arxiv.org/pdf/2508.01579)][[CODE](https://github.com/HHHLF/SECA_master)]
* (AAAI) Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA. [[PDF](https://arxiv.org/pdf/2601.22828)][[CODE](https://github.com/Fazhan-cs/DAC)]
* (AAAI) MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering. [[PDF](https://arxiv.org/pdf/2601.01926)][[CODE](https://github.com/HubuKG/MacVQA)]
* (AAAI) Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models. [[PDF](https://arxiv.org/pdf/2506.12409)]
* (ICLR) Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models. [[PDF](https://arxiv.org/pdf/2510.21175)]
* (ICLR) Fine-tuning MLLMs Without Forgetting Is Easier Than You Think. [[PDF](https://arxiv.org/pdf/2603.14493)]
* (ICLR) Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA. [[PDF](https://arxiv.org/pdf/2412.01004)][[CODE](https://github.com/jeff024/codyra)]
* (ICLR) PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning. [[PDF](https://openreview.net/pdf?id=WdP1NVSzsz)][[CODE](https://github.com/SII-HITclearlove777/PCLR)]
* (CVPR) Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models. [[PDF](https://arxiv.org/pdf/2512.09172)][[CODE](https://github.com/CZZZZZZZZZZZZZZZZZ/PRIMED)]
* (CVPR) ϕ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models. [[PDF](https://arxiv.org/pdf/2602.22601)][[CODE](https://github.com/uark-cviu/FaiDPO)]
* (CVPR) CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning. [[PDF](https://arxiv.org/pdf/2603.02951)]
* (ICML) KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls. [[PDF](https://arxiv.org/pdf/2510.19316)][CODE](https://github.com/KORE-LMM/KORE)]
* (arxiv) Neural Sentinel: Unified Vision Language Model (VLM) for License Plate Recognition with Human-in-the-Loop Continual Learning. [[PDF](https://arxiv.org/pdf/2602.07051)]
* (arxiv) Reinforced Curriculum Pre-Alignment for Domain-Adaptive VLMs. [[PDF](https://arxiv.org/pdf/2602.10740)]
* (arxiv) Continual Learning with Vision-Language Models via Semantic-Geometry Preservation. [[PDF](https://arxiv.org/pdf/2603.12055)]
* (arxiv) Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysis. [[PDF](https://arxiv.org/pdf/2604.00820)][[CODE](https://github.com/XingxingW/CLeaRS-Preview)]
* (arxiv) Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuning. [[PDF](https://arxiv.org/pdf/2604.15705)]
* (arxiv) Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails. [[PDF](https://arxiv.org/pdf/2607.02020)]

### 2025
* (ACL) HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model. [[PDF](https://arxiv.org/pdf/2503.12941)][[CODE](https://github.com/Ghy0501/HiDe-LLaVA)]
* (ACL) Enhancing Multimodal Continual Instruction Tuning with BranchLoRA. [[PDF](https://arxiv.org/pdf/2506.02041)][[CODE](https://github.com/BladeDancer957/BranchLoRA)]
* (ICCV) External Knowledge Injection for CLIP-Based Class-Incremental Learning. [[PDF](https://arxiv.org/pdf/2503.08510)][[CODE](https://github.com/LAMDA-CL/ICCV25-ENGINE)]
* (ICCV) Mind the Gap: Preserving and Compensating for the Modality Gap in CLIP-Based Continual Learning. [[PDF](https://arxiv.org/pdf/2507.09118)][[CODE](https://github.com/RL-VIG/DMNSP)]
* (ICCV) DMNSP: Dynamic Multi-Layer Null Space Projection for Vision-Language Continual Learning. [[PDF](https://openaccess.thecvf.com/content/ICCV2025/html/Kang_Dynamic_Multi-Layer_Null_Space_Projection_for_Vision-Language_Continual_Learning_ICCV_2025_paper.html)][[CODE](https://github.com/RL-VIG/DMNSP)]
* (ICCV) SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning. [[PDF](https://arxiv.org/abs/2411.13949)][[CODE](https://github.com/Minato-Zackie/SMoLoRA)]
* (ICCV) Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models. [[PDF](https://arxiv.org/abs/2508.00260)]
* (ICML) LADA: Scalable Label-Specific CLIP Adapter for Continual Learning. [[PDF](https://arxiv.org/abs/2505.23271)][[CODE](https://github.com/MaolinLuo/LADA)]
* (ICML) SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning. [[PDF](https://arxiv.org/abs/2505.02486)][[CODE](https://github.com/jinpeng0528/SEFE/)]
* (ICML) Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning. [[PDF](https://arxiv.org/abs/2506.11672)][[CODE](https://github.com/gcd19/D-MoLE)]
* (ICLR) C-CLIP: Multimodal Continual Learning for Vision-Language Model. [[PDF](https://openreview.net/pdf?id=sb7qHFYwBc)][[CODE](https://github.com/SmallPigPeppa/C-CLIP)]
* (CVPR) Synthetic Data is an Elegant GIFT for Continual Vision-Language Models. [[PDF](https://arxiv.org/pdf/2503.04229)][[CODE](https://github.com/Luo-Jiaming/GIFT_CL)]
* (CVPR) Language Guided Concept Bottleneck Models for Interpretable Continual Learning. [[PDF](https://arxiv.org/pdf/2503.23283)][[CODE](https://github.com/FisherCats/CLG-CBM)]
* (CVPR) CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering. [[PDF](https://arxiv.org/pdf/2503.00413)][[CODE](https://github.com/ECNU-ICALK/CL-MoE)]
* (EMNLP) ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt. [[PDF](https://arxiv.org/pdf/2410.05849)][[CODE](https://github.com/AuroraZengfh/ModalPrompt)]
* (TPAMI) Learning without Forgetting for Vision-Language Models. [[PDF](https://arxiv.org/pdf/2305.19270)][[CODE](https://github.com/zhoudw-zdw/PROOF/)]
* (TMM) Visual Class Incremental Learning with Textual Priors Guidance based on an Adapted Vision-Language Model. [[PDF](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10897910)][[CODE](https://openi.pcl.ac.cn/OpenMedIA/CIL_Adapterd_VLM)]
* (ECML-PKDD) ChordPrompt: Orchestrating Cross-Modal Prompt Synergy for Multi-Domain Incremental Learning in CLIP. [[PDF](https://arxiv.org/pdf/2506.19608)] [[CODE] (https://github.com/XiaoAI1989/ChordPrompt)]
* (TMLR) Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors. [[PDF](https://arxiv.org/pdf/2505.20680)][[CODE](https://github.com/jeff024/tppt)]
* (TMLR) Efficient Few-Shot Continual Learning in Vision-Language Models. [[PDF](https://arxiv.org/pdf/2502.04098)]
* (EMNLP Findings) Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent. [[PDF](https://arxiv.org/pdf/2502.11740)]
* (ACL Findings) Progressive LoRA for Multimodal Continual Instruction Tuning. [[PDF](https://aclanthology.org/2025.findings-acl.143/)][[CODE](https://github.com/ku-nlp/ProgLoRA)]
* (arXiv) No Images, No Problem: Retaining Knowledge in Continual VQA with Questions-Only Memory. [[PDF](https://arxiv.org/pdf/2502.04469.pdf)][[CODE](https://github.com/IemProg/QUAD)]
* (arxiv) LVP-CLIP: Revisiting CLIP for Continual Learning with Label Vector Pool. [[PDF](https://arxiv.org/pdf/2412.05840)]
* (arxiv) MLLM-CL: Continual Learning for Multimodal Large Language Models. [[PDF](https://arxiv.org/pdf/2506.05453)][[CODE](https://github.com/bjzhb666/MLLM-CL?tab=readme-ov-file)]
* (arXiv) GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning. [[PDF](https://arxiv.org/pdf/2507.19839.pdf)][[CODE](https://github.com/Ppp-Ttt/GNSP)]
* (arxiv) Hierarchical Representation Matching for CLIP-based Class-Incremental Learning. [[PDF](https://arxiv.org/pdf/2509.22645)]
* (arxiv) Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning. [[PDF](https://arxiv.org/pdf/2511.15633)]
* (arxiv) BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning. [[PDF](https://arxiv.org/pdf/2511.11421)]
* (arxiv) Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning. [[PDF](https://arxiv.org/pdf/2511.10974)]
* (arxiv) Representation Calibration and Uncertainty Guidance for Class-Incremental Learning based on Vision Language Model. [[PDF](https://arxiv.org/pdf/2512.09441)]
* (arxiv) Prompt-Based Continual Compositional Zero-Shot Learning. [[PDF](https://arxiv.org/pdf/2512.09172)]
* (arxiv) Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models. [[PDF](https://arxiv.org/pdf/2505.07690)]
* (arxiv) CalFuse: Multi-Modal Continual Learning via Feature Calibration and Parameter Fusion. [[PDF](https://arxiv.org/pdf/2503.18672)]
* (arxiv) Routing-Based Continual Learning for Multimodal Large Language Models. [[PDF](https://arxiv.org/pdf/2511.01831v3)]

### 2024
* (TMLR) Continual Learning in Open-vocabulary Classification with Complementary Memory Systems. [[PDF](https://arxiv.org/pdf/2307.01430)]
* (ICLR) TiC-CLIP: Continual Training of CLIP Models.[[PDF](https://arxiv.org/pdf/2310.16226)][[CODE](https://github.com/apple/ml-tic-clip)]
* (arxiv) A Practitioner’s Guide to Continual Multimodal Pretraining. [[PDF](https://arxiv.org/pdf/2408.14471)][[CODE](https://github.com/ExplainableML/fomo_in_flux/)]
* (CVPR) Generative Multi-modal Models are Good Class-Incremental Learners. [[PDF](https://arxiv.org/pdf/2403.18383)][[CODE](https://github.com/DoubleClass/GMM)]
* (CVPR) Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners. [[PDF](https://arxiv.org/pdf/2404.02117)][[CODE](https://github.com/KU-VGI/PriViLege)]
* (arxiv) Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype. [[PDF](https://arxiv.org/pdf/2408.09984)]
* (AAAI) Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning.[[PDF](https://arxiv.org/pdf/2401.17186)][[CODE](https://github.com/yangbang18/CLFM)]
* (AAAI) Continual Vision-Language Retrieval via Dynamic Knowledge Rectification. [[PDF](https://ojs.aaai.org/index.php/AAAI/article/view/29054)]
* (AAAI) Learning Task-Aware Language-Image Representation for Class-Incremental Object Detection. [[PDF](https://ojs.aaai.org/index.php/AAAI/article/view/28537)]
* (AAAI) GCD: Advancing Vision-Language Models for Incremental Object Detection via Global Alignment and Correspondence Distillation [[PDF](https://ojs.aaai.org/index.php/AAAI/article/view/32864)]
* (ECCV) Anytime Continual Learning for Open Vocabulary Classification. [[PDF](https://arxiv.org/abs/2409.08518v1)] [[CODE](https://github.com/jessemelpolio/AnytimeCL)]
* (ECCV) MagMax: Leveraging Model Merging for Seamless Continual Learning. [[PDF](https://arxiv.org/abs/2407.06322)] [[CODE](https://github.com/danielm1405/magmax)]
* (ECCV) Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models (DIKI). [[PDF](https://arxiv.org/pdf/2407.05342)] [[CODE](https://github.com/lloongx/DIKI)]
* (ECCV) Adapt without Forgetting: Distill Proximity from Dual Teachers in Vision-Language Models. [[PDF](https://www.ecva.net/papers/eccv_2024/papers_ECCV/papers/07052.pdf)]
* (ECCV) Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models. [[PDF](https://arxiv.org/pdf/2403.09296)]
* (ECCV) Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion. [[PDF](https://arxiv.org/pdf/2407.14143)] [[CODE](https://github.com/linlany/RAPF)]
* (NeurIPS) Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models (RAIL). [[PDF](https://arxiv.org/pdf/2406.18868)][[CODE](https://github.com/linghan1997/Regression-based-Analytic-Incremental-Learning)]
* (NeurIPS) CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models. [[PDF](https://arxiv.org/pdf/2403.19137.pdf)][[CODE](https://github.com/srvCodes/clap4clip)]
* (ACM MM) Low-rank Prompt Interaction for Continual Vision-language Retrieval. [[PDF](https://arxiv.org/pdf/2501.14369)][[CODE](https://github.com/Kelvin-ywc/LPI)]
* (ACM MM) Calibrating Prompt from History for Continual Vision-Language Retrieval and Grounding. [[PDF](https://dl.acm.org/doi/10.1145/3664647.3681387)]
* (IJCAI) Continual Multimodal Knowledge Graph Construction. [[PDF](https://arxiv.org/pdf/2305.08698)] [[CODE](https://github.com/zjunlp/ContinueMKGC)]
* (TCSVT) Continual Learning of Image Classes with Language Guidance from a Vision-Language Model. [[PDF](https://ieeexplore.ieee.org/document/10644076/)][[CODE](https://github.com/Fatflower/CIL_LG_VLM)]
* (arXiv) CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning. [[PDF](https://arxiv.org/pdf/2403.10245)]
* (arXiv) LW2G: Learning Whether to Grow for Prompt-based Continual Learning. [[PDF](https://arxiv.org/pdf/2409.18860.pdf)][[CODE](https://github.com/raian08/lw2g)]
* (arXiv) ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy. [[PDF](https://arxiv.org/pdf/2410.10923.pdf)][[CODE](https://github.com/lihong2303/ATLAS)]
* (arxiv) Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning. [[PDF](https://arxiv.org/pdf/2408.01076)]
* (arxiv) CLIP model is an Efficient Online Lifelong Learner. [[PDF](https://arxiv.org/pdf/2405.15155)]

### 2023
* (AAAI) Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task. [[PDF](https://arxiv.org/pdf/2208.12037)] [[CODE](https://github.com/showlab/CLVQA)]
* (CVPR) Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models (ZSCL). [[PDF](https://arxiv.org/abs/2303.06628)] [[CODE](https://github.com/Thunderbeee/ZSCL/tree/main)]
* (CVPR) Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters. [[PDF](https://arxiv.org/abs/2403.11549)] [[CODE](https://github.com/JiazuoYu/MoE-Adapters4CL)]
* (CVPR) VQACL: A Novel Visual Question Answering Continual Learning Setting. [[PDF](https://openaccess.thecvf.com/content/CVPR2023/papers/Zhang_VQACL_A_Novel_Visual_Question_Answering_Continual_Learning_Setting_CVPR_2023_paper.pdf)] [[CODE](https://github.com/zhangxi1997/VQACL)]
* (ICCV) CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation [[PDF](https://arxiv.org/pdf/2308.07146)] [[CODE](https://github.com/KevinLight831/CTP)]
* (ICCV) Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering [[PDF](https://openaccess.thecvf.com/content/ICCV2023/papers/Qian_Decouple_Before_Interact_Multi-Modal_Prompt_Learning_for_Continual_Visual_Question_ICCV_2023_paper.pdf)]
* (ICCVW) Multimodal Parameter-Efficient Few-Shot Class Incremental Learning. [[PDF](https://arxiv.org/pdf/2303.04751)]
* (ICML) Continual vision-language representation learning with off-diagonal information. [[PDF](https://proceedings.mlr.press/v202/ni23c/ni23c.pdf)] [[CODE](https://github.com/Thunderbeee/ZSCL/tree/main)]
* (ACM MM) Multi-Domain Lifelong Visual Question Answering via Self-Critical Distillation. [[PDF](https://dl.acm.org/doi/pdf/10.1145/3581783.3612121)]
* (arXiv) Class Incremental Learning with Pre-trained Vision-Language Models. [[PDF](https://arxiv.org/pdf/2310.20348.pdf)]

### 2022
* (Neurips) S-Prompts Learning with Pre-trained Transformers: An Occam’s Razor for Domain Incremental Learning. [[PDF](https://arxiv.org/abs/2207.12819)] [[CODE](https://github.com/iamwangyabin/S-Prompts)]
* (Neurips) Climb: A continual learning benchmark for vision-and-language tasks. [[PDF](https://proceedings.neurips.cc/paper_files/paper/2022/file/bd3611971089d466ab4ca96a20f7ab13-Paper-Datasets_and_Benchmarks.pdf)] [[CODE](https://github.com/GLAMOR-USC/CLiMB)]
* (ECCV) Generative Negative Text Replay for Continual Vision-Language Pretraining. [[PDF](https://arxiv.org/pdf/2210.17322)] 
* (arxiv) Continual-CLIP: CLIP is an Efficient Continual Learner. [[PDF](https://arxiv.org/abs/2210.03114)] [[CODE](https://github.com/vgthengane/Continual-CLIP/tree/master)]

---

## 🗂️ Datasets & Benchmarks
Evaluating VLM-CL requires specialized benchmarks. As detailed in our survey, the landscape of benchmarks for vision-language continual learning has evolved into three tiers: **Repurposed Unimodal**, **Adapted Multimodal**, and **Native VLM/MLLM-Specific** benchmarks.

The tables below summarize all key datasets used in the literature according to this evolutionary taxonomy.

### 1. Repurposed Unimodal Benchmarks
*Adapted from traditional computer vision datasets. They offer scalability for evaluating catastrophic forgetting, but provide limited insights into cross-modal misalignment.*

### 2. Adapted Multimodal Benchmarks
*Reconfigured from existing multimodal datasets. These preserve cross-modal interactions while introducing continual learning protocols.*

| Dataset | Task Type | Scenario | Modality | Size | Metrics | Link/Source |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **MDL-VQA** | VQA | DIL | Image+Text | 150K | Accuracy | [[Paper](https://dl.acm.org/doi/10.1145/3581783.3612121)] |
| **P9D** | Retrieval | DIL / TIL | Image+Text | 1M+ | AP | [[Paper](https://arxiv.org/abs/2308.07146)] |
| **Flickr30K** | Retrieval | CIL | Image+Annotations | 30K | Accuracy | [[Link](https://shannon.cs.illinois.edu/DenotationGraph/)] |
| **ECommerce-T2I**| Retrieval | CIL | Image+Annotations | 15K | Accuracy | [[Paper](https://dl.acm.org/doi/10.1145/3447548.3467206)] |
| **NExT-QA** | VQA | TIL | Videos+Text | 52K | AP | [[GitHub](https://github.com/doc-doc/NExT-QA)] |

### 3. VLM/MLLM-Specific Benchmarks
*Explicitly designed to address cross-modal forgetting pathologies, zero-shot decay, compositional transfer, and instruction-tuning alignment.*

#### Notes on CL Scenarios:
- **CIL (Class-Incremental Learning)**: New classes are added in sequential tasks.
- **TIL (Task-Incremental Learning)**: Tasks are distinct, and the task identity is known at inference time.
- **DIL (Domain-Incremental Learning)**: The data distribution shifts across tasks (e.g., photos to sketches), but the task itself remains the same.
- **IIL (Instance-Incremental Learning)**: New training instances of previously seen classes appear over time.
- **AIL (Ability-Incremental Learning)**: The model sequentially acquires new capabilities or skills.
- **Time-IL (Time-Incremental Learning)**: Data arrives chronologically, simulating real-world temporal distribution shifts.

---

### 📊 Evaluation Metrics
Evaluating CL in VLMs necessitates a multifaceted approach that addresses both traditional CL desiderata and VLM-specific challenges (e.g., zero-shot generalization and cross-modal alignment). Key metrics discussed in our survey include:

*   **Average & Last Accuracy**: Measures the mean task performance over all learning stages and the retained competence after full training.
*   **Forgetting Ratio & Backward Transfer (BWT)**: Quantifies the maximum performance drop per task post-initial learning, or improvements induced by later learning.
*   **Zero-Shot Transfer & Zero-Shot Degradation (ZSD)**: Explicitly evaluates the model's generalization to unseen tasks and measures the erosion of pre-acquired open-vocabulary capabilities.
*   **Modality-Specific Metrics**: Utilizing Recall@K, mAP, and Answer Precision (AP) for Retrieval and VQA to diagnose asymmetric modality degradation.

Regarding the detailed computation of these evaluation metrics, we provide a clear and intuitive diagram (as shown below) in the paper for comprehensive explanation.

---

## 🚀 Future Directions

Based on the gaps and trends identified in our survey, we propose several key directions for future research:

* 📊 **Unified and Holistic Benchmarking**: Moving beyond simple classification accuracy to incorporate compositional zero-shot evaluation and temporal, web-scale data streams.

* 💬 **Continual Learning for Generative and Interactive Tasks**: Enabling multimodal chatbots and agents to continually learn from human feedback and long-term memory without losing alignment.

* 🤖 **Continual Learning for Vision-Language-Action (VLA) Models**: Transitioning to Embodied AI where cross-modal feature drift precipitates cascading physical failures, requiring skill-compositional strategies.

* 📐 **Towards a Theoretical Understanding**: Developing mathematical models for cross-modal feature drift and theoretical bounds for Parameter-Efficient Adaptation.

---

## Star History

---

## 🤝 How to Contribute
We welcome contributions! If you have a paper, dataset, or codebase that should be on this list, please open a **Pull Request**.
1.  Fork the repository.
2.  Add your resource to the appropriate section in `README.md`. Please maintain the existing format.
3.  Submit a pull request with a clear title and description.

---

## 🌟 Related Resources  
- [Awesome Continual Learning](https://github.com/xialeiliu/Awesome-Incremental-Learning)  
- [Awesome Vision-Language Models](https://github.com/jingyi0000/VLM_survey)

---

## 🛠️ Maintenance
* **Maintainers**: [Yuyang Liu](https://github.com/YuyangSunshine), Qiuhe Hong, [Linlan Huang](https://github.com/linlany), [Alexandra Gomez-Villa](https://github.com/AlexAndorra), and [Dipam Goswami](https://github.com/dipamgoswami).
* **Contact**: sunshineliuyuyang@gmail.com
* *Star the repo to show your support!* ⭐

*Acknowledgments: This repository builds on the incredible efforts of the open-source research community.*

Awesome-Continual-learning-of-Vision-Language-Models

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Awesome-Continual-learning-of-Vision-Language-Models

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README