Abstract:Multi-modal Large Language Models (MLLMs) frequently face challenges from concept drift when dealing with real-world streaming data, wherein distributions change unpredictably. This mainly includes gradual drift due to long-tailed data and sudden drift from Out-Of-Distribution (OOD) data, both of which have increasingly drawn the attention of the research community. While these issues have been extensively studied in the individual domain of vision or language, their impacts on MLLMs in concept drift settings remain largely underexplored. In this paper, we reveal the susceptibility and vulnerability of Vision-Language (VL) models to significant biases arising from gradual drift and sudden drift, particularly in the pre-training. To effectively address these challenges, we propose a unified framework that extends concept drift theory to the multi-modal domain, enhancing the adaptability of the VL model to the distribution unpredictable changes. Additionally, a T-distribution based drift adapter is proposed to effectively mitigate the bias induced by the gradual drift, which also facilitates the model in distinguishing sudden distribution changes through explicit distribution modeling. Extensive experiments demonstrate our method enhances the efficiency and accuracy of image-text alignment in the pre-training of VL models, particularly in the concept drift scenario. Moreover, various downstream tasks exhibit significant improvements in our model's ability to adapt to long-tailed open world. Furthermore, we create a set of multi-modal datasets called OpenMMlo, specifically tailored for the long-tailed open world settings, to validate our findings. To foster the development of the multi-modal community, we have made both OpenMMlo datasets and our code publicly available at: <a class="link-external link-https" href="https://github.com/Anonymous0Knight/ConceptDriftMLLMs" rel="external noopener nofollow">this https URL</a>.

Tracking Concept Drifting with Gaussian Mixture Model

Tracking Concept Drifting with an Online-Optimized Incremental Learning Framework

Tackling Virtual and Real Concept Drifts: An Adaptive Gaussian Mixture Model

Online Semi-Supervised Concept Drift Detection with Density Estimation

Recursive Gaussian Mixture Models for Adaptive Process Monitoring

Online Drift Detection with Maximum Concept Discrepancy

Multi-Channel Adaptive Mixture Background Model for Real-time Tracking.

Concept drift challenge in multimedia anomaly detection: A case study with facial datasets

Concept Drift Detection and Adaptation with Weak Supervision on Streaming Unlabeled Data

Handling Concept Drift in Global Time Series Forecasting

Concept Drift Detection Based on Typicality and Eccentricity

A Novel Incremental Gaussian Mixture Regression and Its Application for Time-varying Multimodal Process Quality Prediction

On the Reliable Detection of Concept Drift from Streaming Unlabeled Data

Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards

Concept Drift Visualization of SVM with Shifting Window

Temporal Attention for Few-Shot Concept Drift Detection in Streaming Data

Drift Detection: Introducing Gaussian Split Detector

Concept Drift Adaptation with Continuous Kernel Learning

An Improved Gaussian Mixture Modeling Algorithm Combining Foreground Matching and Short-Term Stability Measure for Motion Detection

TS-DM: A Time Segmentation-Based Data Stream Learning Method for Concept Drift Adaptation

Concept drift detection methods based on different weighting strategies