Abstract:The high-speed mobile networks offer great potentials to many future intelligent applications, such as autonomous vehicles in smart transportation systems. Such networks provide the possibility to interconnect mobile devices to achieve fast knowledge sharing for efficient collaborative learning and operations, especially with the help of distributed machine learning, e.g., Federated Learning (FL), and modern digital technologies, e.g., Digital Twin (DT) systems. Typically, FL requires a fixed group of participants that have Independent and Identically Distributed (IID) data for accurate and stable model training, which is highly unlikely in real-world mobile network scenarios. In this paper, in order to facilitate the lightweight model training and real-time processing in high-speed mobile networks, we design and introduce an end-edge-cloud structured three-layer Federated Reinforcement Learning (FRL) framework, incorporated with an edge-cloud structured DT system. A dual-Reinforcement Learning (dual-RL) scheme is devised to support optimizations of client node selection and global aggregation frequency during FL via a cooperative decision-making strategy, which is assisted by a two-layer DT system deployed in the edge-cloud for real-time monitoring of mobile devices and environment changes. A model pruning and federated bidirectional distillation (Bi-distillation) mechanism is then developed locally for the lightweight model training, while a model splitting scheme with a lightweight data augmentation mechanism is developed globally to separately optimize the aggregation weights based on a splitted neural network structure (i.e., the encoder and classifier) in a more targeted manner, which can work together to effectively reduce the overall communication cost and improve the non-IID problem. Experiment and evaluation results compared with three baseline methods using two different real-world datasets demonstrate the usefulness and outstanding performance of our proposed FRL model in communication-efficient model training and non-IID issue alleviation for high-speed mobile network scenarios.

Distributed Deep Reinforcement Learning for Resource Allocation in Digital Twin Networks

Parallel Digital Twin-driven Deep Reinforcement Learning for User Association and Load Balancing in Dynamic Wireless Networks

Digital Twin-Enhanced Deep Reinforcement Learning for Resource Management in Networks Slicing

Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

Deep Reinforcement Learning for Resource Demand Prediction and Virtual Function Network Migration in Digital Twin Network

Federated Deep Reinforcement Learning for Task Offloading in Digital Twin Edge Networks

Double Deep Q-Network Based Distributed Resource Matching Algorithm for D2D Communication

Collective reinforcement learning based resource allocation for digital twin service in 6G networks

Deep Multi-Agent Reinforcement Learning for Resource Allocation in D2D Communication Underlaying Cellular Networks

Deep Reinforcement Learning for Online Resource Allocation in IoT Networks: Technology, Development, and Future Challenges

Deep Reinforcement Learning for Dynamic Clustering and Resource Allocation in Smart-Duplex Networks

Two-Timescale Digital Twin Assisted Model Interference and Retraining over Wireless Network

Resource Allocation in OFDMA Networks with Deep Reinforcement Learning

Distributed Two-tier DRL Framework for Cell-Free Network: Association, Beamforming and Power Allocation.

Digital Twin Enhanced Federated Reinforcement Learning with Lightweight Knowledge Distillation in Mobile Networks

Dynamic Clustering and Resource Allocation Using Deep Reinforcement Learning for Smart-Duplex Networks

Hybrid Centralized-Distributed Resource Allocation Based on Deep Reinforcement Learning for Cooperative D2D Communications

Federated Policy Distillation for Digital Twin-Enabled Intelligent Resource Trading in 5G Network Slicing

Deep Reinforcement Learning for Joint Channel Selection and Power Control in D2D Networks

Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network