Abstract:Generative foundation models are facilitating significant transformations in the field of artificial intelligence.They demonstrate general artificial intelligence in diverse research fields,including natural language processing,multi-modal content understanding,imagery,and multimodal content synthesis.Generative foundation models often consist of billions or even hundreds of billions of parameters.Thus,they are often deployed on the cloud side to provide powerful and general intelligent services.However,this type of service can be confronted with crucial challenges in practice,such as high latency induced by communications between the cloud and local devices,and insufficient personalization capabilities due to the fact that servers often do not have access to local data considering privacy concerns.By contrast,low-complexity lightweight models are located at the edge side to capture personalized and dynamic scenario data.However,they may suf-fer from poor generalization.Large and lightweight(or large-small)model collaboration aims to integrate the general intelli-gence of large foundation models and the personalized intelligence of small lightweight models.This integration empowers downstream vertical domain-specific applications through the interaction and collaboration of both types of intelligent mod-els.Large and small model collaboration has recently attracted increasing attention and becomes the focus of research and development in academia and industry.It has also been predicted to be an important trend in technology.We therefore try to thoroughly investigate this area by highlighting recent progress and bringing potential inspirations for related research.In this study,we first overview representative large language models(LLMs)and large multimodal models.We focus on their mainstream Transformer-based model architectures including encoder-only,decoder-only,and encoder-decoder models.Corresponding pre-training technologies such as next sentence prediction,sequence-to-sequence modeling,contrastive learning,and parameter-efficient fine-tuning methods with representatives including low-rank adaptation and prompt tuning are also explored.We then review the development history and the latest advancement of model compression techniques,including model pruning,model quantization,and knowledge distillation in the era of foundation models.Based on the dif-ferences in terms of model collaboration purposes and mechanisms,we propose a new classification method and taxonomies for the large-small model collaboration study,namely,collaborative training,collaborative inference,and collaborative planning.Specifically,we summarize recent and representative methods that consist of dual-directional knowledge distilla-tion between large models at the cloud side and small models deployed at the edge side,modular design of intelligent mod-els that split functional models between the cloud and edge,and generative agents that collaborate to complete more com-plex tasks in an autonomous and intelligent manner.In collaborative training,a main challenge is dealing with the hetero-geneity in data distribution and model architectures between the cloud and client sides.Data privacy may also be a concern during collaborative training,particularly in privacy sensitive cases.Despite much progress in collaborative inference,slic-ing and completing a complicated task in a collective way automatically remain challenging.Furthermore,the communica-tion costs between computing facilities might be another concern.Collective planning is a new paradigm that gains attention with the increasing study and promising progress of LLM-centric agents(LLM agents).This paradigm often involves mul-tiple LLM agents who compete or cooperate together to complete a challenging task.It often leverages emerging capabilities such as in-context learning and chain-of-thoughts of LLMs to automatically dive a complicated task into several subtasks.By completing and assembling different subtasks,the global task can be conducted in a collaborative manner.This scheme finds diverse applications such as developing games and simulating social societies.However,it may suffer from drawbacks inherent in LLMs,including hallucination and adversarial vulnerabilities.Thus,more robust and reliable collaborative planning schemes remain to be investigated.In summary,this work surveys the large-small model collaboration techniques from the perspectives of generative foundation models,model compression,and heterogeneous model collaboration via LLM agents.This work also compares the advantages and disadvantages between international and domestic technology developments in this research realm.We conclude that,although the gaps are narrowing between domestic and advanced international studies in this area,particularly for newly emerging LLM agents,we may still lack original and major break-throughs.Certain notable advantages of domestic progress are closely related to industrial applications due to its rich data resources from industries.Therefore,the development of domain specific LLMs is advanced.In addition,this study envi-sions the applications of large-small model collaboration and discusses certain key challenges and promising directions in this topic.1)The design of efficient model architectures includes developing new model architectures that can achieve low-complexity inference speed while maintaining efficient long-sequence modeling abilities as Transformers and further improv-ing the scalability of mixture-of-expert-based architectures.2)Current model compression methods are mainly designed for vision models.Thus,developing techniques specifically for LLMs and large multimodal models is important to preserve their emergent abilities during compression.3)Existing personalization methods specially focus on discriminative models,and due attention needs to be paid for efficient personalization for generative foundation models.4)Generative intelligence often suffers from fraudulent contents(e.g.,generated fake imagery,deepfake videos,and fake news)and different types of attacks(e.g.,adversarial attacks,the jailing breaking attacks,and the Byzantine attacks).Thus,security and trust-worthy issues arise in their practical applications.Therefore,this study also advocates a deeper investigation of these emerging security threats.Then,it develops effective defenses accordingly to countermeasure these crucial issues during large-small model collaboration for empowering vertical domains more safely.

Federated Learning in Big Model Era: Domain-Specific Multimodal Large Models

Federated Large Language Model: Solutions, Challenges and Future Directions

Research on data fusion method based on Federated learning

Large-Scale Model Meets Federated Learning: A Hierarchical Hybrid Distributed Training Mechanism for Intelligent Intersection Large-Scale Model

Federated Large Language Models: Current Progress and Future Directions

Multimodal federated learning: Concept, methods, applications and future directions

Multimodal Federated Learning on IoT Data

Multimodal Fusion with Block Term Decomposition for Asynchronous Federated Learning

Multimodal Federated Learning

A survey of multimodal federated learning: background, applications, and perspectives

Integration of Large Language Models and Federated Learning

A unified framework for multi-modal federated learning

Optimizing Multimodal Federated Learning: Novel Approaches for Efficient Model Aggregation and Client Sampling

Federated Learning Framework Based on Data Value Evaluation in Industrial IoT

Evaluation Framework For Large-scale Federated Learning

Federated Learning for IoT Devices with Domain Generalization

Towards Federated Large Language Models: Motivations, Methods, and Future Directions

Advances in Edge-Cloud Collaboration and Evolution for Large-Small Models

Collaborative Multimodal Vehicular Transformer Training Using Federated Learning

Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaboration

The Future of Large Language Model Pre-training is Federated