Abstract:Deep learning-based Autonomous Driving (AD) models often exhibit poor generalization due to data heterogeneity in an ever domain-shifting environment. While Federated Learning (FL) could improve the generalization of an AD model (known as FedAD system), conventional models often struggle with under-fitting as the amount of accumulated training data progressively increases. To address this issue, instead of conventional small models, employing Large Vision Models (LVMs) in FedAD is a viable option for better learning of representations from a vast volume of data. However, implementing LVMs in FedAD introduces three challenges: (I) the extremely high communication overheads associated with transmitting LVMs between participating vehicles and a central server; (II) lack of computing resource to deploy LVMs on each vehicle; (III) the performance drop due to LVM focusing on shared features but overlooking local vehicle characteristics. To overcome these challenges, we propose pFedLVM, a LVM-Driven, Latent Feature-Based Personalized Federated Learning framework. In this approach, the LVM is deployed only on central server, which effectively alleviates the computational burden on individual vehicles. Furthermore, the exchange between central server and vehicles are the learned features rather than the LVM parameters, which significantly reduces communication overhead. In addition, we utilize both shared features from all participating vehicles and individual characteristics from each vehicle to establish a personalized learning mechanism. This enables each vehicle's model to learn features from others while preserving its personalized characteristics, thereby outperforming globally shared models trained in general FL. Extensive experiments demonstrate that pFedLVM outperforms the existing state-of-the-art approaches.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - scale vision models (LVMs) to improve the model generalization ability of autonomous driving systems (AD) in the federated learning framework, while overcoming challenges such as computational resources, communication overhead, and preservation of personalized features when using large - scale models in federated learning. Specifically, the paper points out: 1. **Data heterogeneity and domain shift**: Deep - learning models in autonomous driving usually have poor generalization ability due to data heterogeneity and environmental changes. Although federated learning (FL) can improve the model generalization ability by combining data from multiple vehicles, as the amount of training data increases, traditional models are prone to under - fitting problems. 2. **Advantages and challenges of large - scale vision models**: The introduction of large - scale vision models (LVMs) can better learn representations from a large amount of data, thus solving the under - fitting problem. However, deploying LVMs in the federated learning environment faces the following three main challenges: - **High communication overhead**: LVMs have a large number of parameters, and transmitting these parameters will lead to extremely high communication costs. - **Insufficient computational resources**: The computing resources on the vehicle side are limited, and it is difficult to support the local training of LVMs. - **Ignoring local features**: LVMs tend to extract shared features and may ignore the unique local features of each vehicle, resulting in performance degradation. To solve these problems, the authors propose the pFedLVM framework, whose core idea is: - **Deploy LVM on the central server**: LVM only runs on the central server, reducing the computational burden on the vehicle side, and significantly reducing the communication overhead by exchanging features instead of model parameters. - **Personalized federated learning mechanism**: Combine the shared features of all vehicles and the local features of each vehicle to establish a personalized learning mechanism, ensuring that the model of each vehicle can learn knowledge from other vehicles and maintain its own uniqueness. Through this method, pFedLVM not only improves the generalization ability of the model, but also significantly outperforms the existing state - of - the - art methods in multiple evaluation indicators, specifically, mIoU, mF1, mPrecision, and mRecall are increased by 18.47%, 25.60%, 51.03%, and 14.19% respectively.

pFedLVM: A Large Vision Model (LVM)-Driven and Latent Feature-Based Personalized Federated Learning Framework in Autonomous Driving

FedRAV: Hierarchically Federated Region-Learning for Traffic Object Classification of Autonomous Vehicles

FedVisual: Heterogeneity-Aware Model Aggregation for Federated Learning in Visual-Based Vehicular Crowdsensing

Federated Deep Learning Meets Autonomous Vehicle Perception: Design and Verification

Federated Learning with Heterogeneous Data Handling for Robust Vehicular Object Detection

On the Federated Learning Framework for Cooperative Perception

Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Driving

Advance-FL: A3C-based Adaptive Asynchronous Online Federated Learning for Vehicular Edge Cloud Computing Networks

Deep Federated Learning for Autonomous Driving

Adaptive and Parallel Split Federated Learning in Vehicular Edge Computing

Federated Large Language Model: Solutions, Challenges and Future Directions

FedPAW: Federated Learning with Personalized Aggregation Weights for Urban Vehicle Speed Prediction

FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneity

Privacy-Preserved Federated Learning for Autonomous Driving

Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Data

Federated Transfer-Ordered-Personalized Learning for Driver Monitoring Application

Risk-Aware Reinforcement Learning Based Federated Learning Framework for Io V

Asynchronous Federated Learning for Edge-assisted Vehicular Networks

Mobility, Communication and Computation Aware Federated Learning for Internet of Vehicles

Beyond ADMM: A Unified Client-variance-reduced Adaptive Federated Learning Framework

FedPylot: Navigating Federated Learning for Real-Time Object Detection in Internet of Vehicles