Abstract:This paper introduces a full solution for decentralized routing in Low Earth Orbit satellite constellations based on continual Deep Reinforcement Learning (DRL). This requires addressing multiple challenges, including the partial knowledge at the satellites and their continuous movement, and the time-varying sources of uncertainty in the system, such as traffic, communication links, or communication buffers. We follow a multi-agent approach, where each satellite acts as an independent decision-making agent, while acquiring a limited knowledge of the environment based on the feedback received from the nearby agents. The solution is divided into two phases. First, an offline learning phase relies on decentralized decisions and a global Deep Neural Network (DNN) trained with global experiences. Then, the online phase with local, on-board, and pre-trained DNNs requires continual learning to evolve with the environment, which can be done in two different ways: (1) Model anticipation, where the predictable conditions of the constellation are exploited by each satellite sharing local model with the next satellite; and (2) Federated Learning (FL), where each agent's model is merged first at the cluster level and then aggregated in a global Parameter Server. The results show that, without high congestion, the proposed Multi-Agent DRL framework achieves the same E2E performance as a shortest-path solution, but the latter assumes intensive communication overhead for real-time network-wise knowledge of the system at a centralized node, whereas ours only requires limited feedback exchange among first neighbour satellites. Importantly, our solution adapts well to congestion conditions and exploits less loaded paths. Moreover, the divergence of models over time is easily tackled by the synergy between anticipation, applied in short-term alignment, and FL, utilized for long-term alignment.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve decentralized routing in low - Earth - orbit satellite constellations (LSatCs). Specifically, the paper focuses on how, in a dynamic and partially observable environment, each satellite can make routing decisions independently as an independent decision - making agent based on local observations and limited feedback through the continuous deep reinforcement learning (DRL) method. This involves the following key challenges: 1. **Dynamic Topology**: Satellites are constantly moving, resulting in frequent changes in network topology and frequent link disruptions. 2. **High Propagation Delay**: Due to the large distance between satellites and ground stations, the signal propagation delay is high. 3. **Dynamic and Unpredictable Traffic**: The traffic injected from the ground is dynamic, unbalanced, and difficult to predict. 4. **Resource Constraints**: The computing and communication resources on satellites are limited, and the connection with ground infrastructure is also limited. 5. **Uncertainty**: Sources of uncertainty in the system include changes in links, traffic, and buffer states. To solve these problems, the paper proposes a multi - agent continuous learning framework that combines offline global path calculation and online distributed model updates. The specific methods are as follows: - **Offline Learning Phase**: Use a global deep neural network (DNN) for training to learn the optimal path at each possible location and congestion level. - **Online Learning Phase**: Each satellite uses a pre - trained DNN model to achieve continuous learning in two ways: - **Model Anticipation**: Utilizing the predictability of constellation movement, each satellite shares its Q - network with satellites behind it in the orbit to reduce signaling overhead. - **Federated Learning (FL)**: Each satellite continuously trains the pre - loaded model on its local data set, periodically aggregates the target network at the cluster level, and then performs global aggregation at the parameter server (PS). The main contribution of the paper is to propose a continuous learning framework that can adapt to environmental changes, enabling satellites to make routing decisions independently in a dynamic environment without relying on the global knowledge of a central node. In this way, the paper provides a method for achieving efficient and adaptive routing in resource - constrained satellite networks.

Continual Deep Reinforcement Learning for Decentralized Satellite Routing

Multi-Agent Deep Reinforcement Learning for Distributed Satellite Routing

Multi-Commodity Flow Routing for Large-Scale LEO Satellite Networks Using Deep Reinforcement Learning

Deep Reinforcement Learning-Based Autonomous Mission Planning Method for High and Low Orbit Multiple Agile Earth Observing Satellites

Enabling High-Throughput Routing for LEO Satellite Broadband Networks: A Flow-Centric Deep Reinforcement Learning Approach

An open source Multi-Agent Deep Reinforcement Learning Routing Simulator for satellite networks

Fast-Convergence Reinforcement Learning for Routing in LEO Satellite Networks

Reinforcement Learning Based Intelligent Routing for Software Defined LEO Satellite Networks

Spatial Location Aided Fully-Distributed Dynamic Routing for Large-Scale LEO Satellite Networks

Q-learning for distributed routing in LEO satellite constellations

SDDRL-SR: A High-Reliability Satellite Routing Algorithm Based on Deep Reinforcement Learning

Deep Reinforcement Learning for Congestion Control and Routing Optimization in LEO Satellite Networks

A DRL Empowered Multipath Cooperative Routing for Ultra-Dense LEO Satellite Networks

Inter-Satellite Routing for LEO Satellite Networks: A GNN and DRL Integrated Approach

Shaping Rewards, Shaping Routes: On Multi-Agent Deep Q-Networks for Routing in Satellite Constellation Networks

An Intelligent Routing Algorithm for LEO Satellites Based on Deep Reinforcement Learning

Distributed Routing Algorithm for LEO Satellite Network Based on Deep Reinforcement Learning

A robust routing strategy based on deep reinforcement learning for mega satellite constellations

Fully-Distributed Dynamic Packet Routing for LEO Satellite Networks: A GNN-Enhanced Multi-Agent Reinforcement Learning Approach

Graph Neural Network and Reinforcement Learning Based Routing for Mega LEO Satellite Constellations

Collaborative Computing in Non-Terrestrial Networks: A Multi-Time-Scale Deep Reinforcement Learning Approach