Abstract:The burgeoning field of Multimodal Large Language Models (MLLMs) has exhibited remarkable performance in diverse tasks such as captioning, commonsense reasoning, and visual scene understanding. However, the deployment of these large-scale MLLMs on client devices is hindered by their extensive model parameters, leading to a notable decline in generalization capabilities when these models are compressed for device deployment. Addressing this challenge, we introduce a Cloud-Device Collaborative Continual Adaptation framework, designed to enhance the performance of compressed, device-deployed MLLMs by leveraging the robust capabilities of cloud-based, larger-scale MLLMs. Our framework is structured into three key components: a device-to-cloud uplink for efficient data transmission, cloud-based knowledge adaptation, and an optimized cloud-to-device downlink for model deployment. In the uplink phase, we employ an Uncertainty-guided Token Sampling (UTS) strategy to effectively filter out-of-distribution tokens, thereby reducing transmission costs and improving training efficiency. On the cloud side, we propose Adapter-based Knowledge Distillation (AKD) method to transfer refined knowledge from large-scale to compressed, pocket-size MLLMs. Furthermore, we propose a Dynamic Weight update Compression (DWC) strategy for the downlink, which adaptively selects and quantizes updated weight parameters, enhancing transmission efficiency and reducing the representational disparity between cloud and device models. Extensive experiments on several multimodal benchmarks demonstrate the superiority of our proposed framework over prior Knowledge Distillation and device-cloud collaboration methods. Notably, we also validate the feasibility of our approach to real-world experiments.

Hybrid SLM and LLM for Edge-Cloud Collaborative Inference

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing

Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance

CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Efficient Deployment of Large Language Model Across Cloud-Device Systems

PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services

Adaptive Layer Splitting for Wireless LLM Inference in Edge Computing: A Model-Based Reinforcement Learning Approach

EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

Efficient and Economic Large Language Model Inference with Attention Offloading

LLMCad: Fast and Scalable On-device Large Language Model Inference

Hybrid SD: Edge-Cloud Collaborative Inference for Stable Diffusion Models

Mobile Edge Intelligence for Large Language Models: A Contemporary Survey

AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality

ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation

Multi-Compression Scale DNN Inference Acceleration based on Cloud-Edge-End Collaboration

Collaborative Inference for Large Models with Task Offloading and Early Exiting

Cached Model-as-a-Resource: Provisioning Large Language Model Agents for Edge Intelligence in Space-air-ground Integrated Networks

Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM

Cloud-Device Collaborative Learning for Multimodal Large Language Models