Abstract:Vision Language Models (VLMs), pre-trained on large-scale image-text datasets, enable zero-shot predictions for unseen data but may underperform on specific unseen tasks. Continual learning (CL) can help VLMs effectively adapt to new data distributions without joint training, but faces challenges of catastrophic forgetting and generalization forgetting. Although significant progress has been achieved by distillation-based methods, they exhibit two severe limitations. One is the popularly adopted single-teacher paradigm fails to impart comprehensive knowledge, The other is the existing methods inadequately leverage the multimodal information in the original training dataset, instead they rely on additional data for distillation, which increases computational and storage overhead. To mitigate both limitations, by drawing on Knowledge Integration Theory (KIT), we propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. During the four stages, we first leverage prototypes to align across modalities, eliciting cross-modal knowledge, then adding new knowledge by constructing fine-grained intra- and inter-modality relationships with prototypes. After that, knowledge from two teacher models is adaptively distinguished and re-weighted. Finally, we connect between models from intra- and inter-task, integrating preceding and new knowledge. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks, showcasing its potential in adapting VLMs to evolving data distributions.

Continual Learning of Image Classes with Language Guidance from a Vision-Language Model

Learning without Forgetting for Vision-Language Models

Don't Stop Learning: Towards Continual Learning for the CLIP Model

LVP-CLIP:Revisiting CLIP for Continual Learning with Label Vector Pool

Continually Learn to Map Visual Concepts to Large Language Models in Resource-constrained Environments

Class Incremental Learning with Pre-trained Vision-Language Models

Enhancing Visual Continual Learning with Language-Guided Supervision

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

Fine-Grained Visual Prompt Learning of Vision-Language Models for Image Recognition

Advancing Cross-domain Discriminability in Continual Learning of Vison-Language Models

Continual Pre-Training Mitigates Forgetting in Language and Vision

Continual Learning in Open-vocabulary Classification with Complementary Memory Systems

CogVLM: Visual Expert for Pretrained Language Models

Continual Vision-Language Retrieval Via Dynamic Knowledge Rectification

Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models

CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning

Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models

Interactive Continual Learning: Fast and Slow Thinking

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Generative Negative Text Replay for Continual Vision-Language Pretraining