Abstract:This paper introduces a full solution for decentralized routing in Low Earth Orbit satellite constellations based on continual Deep Reinforcement Learning (DRL). This requires addressing multiple challenges, including the partial knowledge at the satellites and their continuous movement, and the time-varying sources of uncertainty in the system, such as traffic, communication links, or communication buffers. We follow a multi-agent approach, where each satellite acts as an independent decision-making agent, while acquiring a limited knowledge of the environment based on the feedback received from the nearby agents. The solution is divided into two phases. First, an offline learning phase relies on decentralized decisions and a global Deep Neural Network (DNN) trained with global experiences. Then, the online phase with local, on-board, and pre-trained DNNs requires continual learning to evolve with the environment, which can be done in two different ways: (1) Model anticipation, where the predictable conditions of the constellation are exploited by each satellite sharing local model with the next satellite; and (2) Federated Learning (FL), where each agent's model is merged first at the cluster level and then aggregated in a global Parameter Server. The results show that, without high congestion, the proposed Multi-Agent DRL framework achieves the same E2E performance as a shortest-path solution, but the latter assumes intensive communication overhead for real-time network-wise knowledge of the system at a centralized node, whereas ours only requires limited feedback exchange among first neighbour satellites. Importantly, our solution adapts well to congestion conditions and exploits less loaded paths. Moreover, the divergence of models over time is easily tackled by the synergy between anticipation, applied in short-term alignment, and FL, utilized for long-term alignment.

A robust routing strategy based on deep reinforcement learning for mega satellite constellations

Multi-Commodity Flow Routing for Large-Scale LEO Satellite Networks Using Deep Reinforcement Learning

DRL-ER: An Intelligent Energy-Aware Routing Protocol With Guaranteed Delay Bounds in Satellite Mega-Constellations

Reinforcement learning based dynamic distributed routing scheme for mega LEO satellite networks

Graph Neural Network and Reinforcement Learning Based Routing for Mega LEO Satellite Constellations

A DRL Empowered Multipath Cooperative Routing for Ultra-Dense LEO Satellite Networks

Deep Reinforcement Learning-Based Multipath Routing for LEO Megaconstellation Networks

An Intelligent Routing Algorithm for LEO Satellites Based on Deep Reinforcement Learning

Reinforcement Learning Based Intelligent Routing for Software Defined LEO Satellite Networks

Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approach

On An Intelligent Hierarchical Routing Strategy for Ultra-Dense Free Space Optical Low Earth Orbit Satellite Networks

An Intelligent Area-segmentation Enabled Hybrid Routing Method in Mega-constellations.

Robust Beam-to-Satellite Routing Strategies for Megaconstellations

SDDRL-SR: A High-Reliability Satellite Routing Algorithm Based on Deep Reinforcement Learning

Deep Reinforcement Learning for Congestion Control and Routing Optimization in LEO Satellite Networks

Continual Deep Reinforcement Learning for Decentralized Satellite Routing

Enabling High-Throughput Routing for LEO Satellite Broadband Networks: A Flow-Centric Deep Reinforcement Learning Approach

Distributed Routing Algorithm for LEO Satellite Network Based on Deep Reinforcement Learning

A Distributed Survivable Routing Algorithm for Mega-Constellations With Inclined Orbits

A Topology Design Method for Satellite Networks Based on Deep Reinforcement Learning

Spatial Location Aided Fully-Distributed Dynamic Routing for Large-Scale LEO Satellite Networks