Abstract:Roadside unit (RSU) can significantly improve the safety and robustness of autonomous vehicles through Vehicle-to-Everything (V2X) communication. Currently, the usage of a single RSU mainly focuses on real-time inference and V2X collaboration, while neglecting the potential value of the high-quality data collected by RSU sensors. Integrating the vast amounts of data from numerous RSUs can provide a rich source of data for model training. However, the absence of ground truth annotations and the difficulty of transmitting enormous volumes of data are two inevitable barriers to fully exploiting this hidden value. In this paper, we introduce FedRSU, an innovative federated learning framework for self-supervised scene flow estimation. In FedRSU, we present a recurrent self-supervision training paradigm, where for each RSU, the scene flow prediction of points at every timestamp can be supervised by its subsequent future multi-modality observation. Another key component of FedRSU is federated learning, where multiple devices collaboratively train an ML model while keeping the training data local and private. With the power of the recurrent self-supervised learning paradigm, FL is able to leverage innumerable underutilized data from RSU. To verify the FedRSU framework, we construct a large-scale multi-modality dataset RSU-SF. The dataset consists of 17 RSU clients, covering various scenarios, modalities, and sensor settings. Based on RSU-SF, we show that FedRSU can greatly improve model performance in ITS and provide a comprehensive benchmark under diverse FL scenarios. To the best of our knowledge, we provide the first real-world LiDAR-camera multi-modal dataset and benchmark for the FL community.

What problem does this paper attempt to address?

The paper primarily addresses the application of Roadside Units (RSUs) in Intelligent Transportation Systems (ITS) and proposes a federated learning framework named FedRSU to solve the following two key issues: 1. **Data Annotation Challenge**: The traditional traffic data annotation process is very time-consuming and costly, making it difficult to annotate all the data collected by RSUs to continuously optimize perception models. 2. **Data Integration Challenge**: Integrating data or models from multiple RSUs is not straightforward. Traffic scene data may contain sensitive information and is voluminous, making centralized model training challenging. To address these issues, the paper proposes the FedRSU framework, which combines two major technologies: self-supervised learning and federated learning. - **Self-Supervised Learning**: By leveraging continuous sensor data streams, the paper proposes a cyclic self-supervised training paradigm. In this paradigm, the model predictions on each RSU can be supervised by subsequent multimodal observational data, enabling continuous learning and improvement of the model without the need for manual annotation. - **Federated Learning**: To overcome the data integration challenge, the paper introduces federated learning to allow multiple RSUs to collaboratively train models without sharing raw data. This approach protects data privacy and effectively utilizes a large amount of underutilized data. Additionally, the paper constructs a large-scale multimodal dataset named RSU-SF, which includes data collected from 17 RSUs and additional data from 4 onboard clients. This dataset covers various scenarios, modalities, and sensor setups to validate the effectiveness of the FedRSU framework. Experimental results show that under different federated learning settings, FedRSU can significantly improve model performance and has good generalization capabilities. Furthermore, the paper demonstrates that FedRSU is a general system compatible with various existing self-supervised scene flow methods and federated learning technologies. In summary, the goal of FedRSU is to address the issues of data annotation and data integration in intelligent transportation systems through an innovative federated learning framework and self-supervised learning strategies, thereby enhancing the accuracy and reliability of RSU perception systems.

FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

FedRC: A Rapid-Converged Hierarchical Federated Learning Framework in Street Scene Semantic Understanding

FedRAV: Hierarchically Federated Region-Learning for Traffic Object Classification of Autonomous Vehicles

FedVisual: Heterogeneity-Aware Model Aggregation for Federated Learning in Visual-Based Vehicular Crowdsensing

Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentation

FedRS-Net: A Federated Learning Approach for Collaborative Multi-Modal Maritime Analytics

Scene adaptation in adverse conditions: a multi-sensor fusion framework for roadside traffic perception

Self-Supervised Multi-Frame Neural Scene Flow

Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Driving

SeFlow: A Self-Supervised Scene Flow Method in Autonomous Driving

RS-DFM: A Remote Sensing Distributed Foundation Model for Diverse Downstream Tasks

Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works

Few-shot remote sensing image scene classification: Recent advances, new baselines, and future trends

Federated Self-Supervised Learning of Monocular Depth Estimators for Autonomous Vehicles

ORSI Salient Object Detection via Progressive Semantic Flow and Uncertainty-Aware Refinement

DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR

Federated Deep Learning Meets Autonomous Vehicle Perception: Design and Verification

Federated Learning for Vehicle Trajectory Prediction: Methodology and Benchmark Study

FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving

Federated Learning for Traffic Flow Prediction with Synthetic Data Augmentation