Joint Computation Offloading and Resource Allocation for LEO Satellite Networks Using Hierarchical Multi-Agent Reinforcement Learning

Junyu Lai,Huashuo Liu,Guoyao Xu,Weiwei Jiang,Xiong Wang,Dingde Jiang

DOI: https://doi.org/10.1109/tccn.2024.3510562

IF: 6.359

2024-01-01

IEEE Transactions on Cognitive Communications and Networking

Abstract:The integration of edge computing with LEO satellite broadband networks (LSBNs) offers a transformative potential, yet remains underexplored in the optimization of joint computation offloading and resource allocation (JCORA). This problem is compounded by the issues of load imbalance and hybrid action spaces. To tackle them, we firstly propose a multi-level edge computing architecture that leverages the inter-satellite links to enable collaborative offloading among neighboring satellites, thereby enhancing global resource utilization and load balance in LSBNs. Moreover, existing studies demonstrate the benefits of deep reinforcement learning (DRL) for JCORA optimization but struggle with the complexities of hybrid action spaces. To address this, we elaborate a novel hierarchical multi-agent DRL (HMADRL) framework that decomposes the JCORA problem into two-layered subproblems, namely global computation offloading and local resource allocation. This decomposition effectively mitigates the challenge posed by hybrid action spaces. The computation offloading subproblem is formulated as a delayed-reward partially observable Markov decision process, optimized by using multi-agent deep Q-networks specialized in discrete action outputs. Meanwhile, the resource allocation subproblem is addressed through the deep deterministic policy gradient model, adept at handling continuous actions. Extensive experiments validate our approach, demonstrating improvements in delay reduction, outrage rate, and load balancing compared to baselines.