Abstract:The burgeoning field of Multimodal Large Language Models (MLLMs) has exhibited remarkable performance in diverse tasks such as captioning, commonsense reasoning, and visual scene understanding. However, the deployment of these large-scale MLLMs on client devices is hindered by their extensive model parameters, leading to a notable decline in generalization capabilities when these models are compressed for device deployment. Addressing this challenge, we introduce a Cloud-Device Collaborative Continual Adaptation framework, designed to enhance the performance of compressed, device-deployed MLLMs by leveraging the robust capabilities of cloud-based, larger-scale MLLMs. Our framework is structured into three key components: a device-to-cloud uplink for efficient data transmission, cloud-based knowledge adaptation, and an optimized cloud-to-device downlink for model deployment. In the uplink phase, we employ an Uncertainty-guided Token Sampling (UTS) strategy to effectively filter out-of-distribution tokens, thereby reducing transmission costs and improving training efficiency. On the cloud side, we propose Adapter-based Knowledge Distillation (AKD) method to transfer refined knowledge from large-scale to compressed, pocket-size MLLMs. Furthermore, we propose a Dynamic Weight update Compression (DWC) strategy for the downlink, which adaptively selects and quantizes updated weight parameters, enhancing transmission efficiency and reducing the representational disparity between cloud and device models. Extensive experiments on several multimodal benchmarks demonstrate the superiority of our proposed framework over prior Knowledge Distillation and device-cloud collaboration methods. Notably, we also validate the feasibility of our approach to real-world experiments.

On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists

Collaborative Learning Between Cloud and End Devices

Learning to Decode Collaboratively with Multiple Language Models

Personalized Collaborative Fine-Tuning for On-Device Large Language Models

FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts

How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning

MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

Cloud-Device Collaborative Learning for Multimodal Large Language Models

Federated Multi-Task Learning under a Mixture of Distributions

Learning to Collaborate in Decentralized Learning of Personalized Models

Federated Mutual Learning: a Collaborative Machine Learning Method for Heterogeneous Data, Models, and Objectives

Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer

Collaborative Machine Learning Model Building with Families Using Co-ML

CoLLM: A Collaborative LLM Inference Framework for Resource-Constrained Devices

CCoE: A Compact LLM with Collaboration of Experts

Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models

Mixture-of-Experts Learner for Single Long-Tailed Domain Generalization

Co-ML: Collaborative Machine Learning Model Building for Developing Dataset Design Practices