Age-Based Scheduling for Mobile Edge Computing: A Deep Reinforcement Learning Approach

Xingqiu He,Chaoqun You,Tony Q. S. Quek

2024-02-23

Abstract:With the rapid development of Mobile Edge Computing (MEC), various real-time applications have been deployed to benefit people's daily lives. The performance of these applications relies heavily on the freshness of collected environmental information, which can be quantified by its Age of Information (AoI). In the traditional definition of AoI, it is assumed that the status information can be actively sampled and directly used. However, for many MEC-enabled applications, the desired status information is updated in an event-driven manner and necessitates data processing. To better serve these applications, we propose a new definition of AoI and, based on the redefined AoI, we formulate an online AoI minimization problem for MEC systems. Notably, the problem can be interpreted as a Markov Decision Process (MDP), thus enabling its solution through Reinforcement Learning (RL) algorithms. Nevertheless, the traditional RL algorithms are designed for MDPs with completely unknown system dynamics and hence usually suffer long convergence times. To accelerate the learning process, we introduce Post-Decision States (PDSs) to exploit the partial knowledge of the system's dynamics. We also combine PDSs with deep RL to further improve the algorithm's applicability, scalability, and robustness. Numerical results demonstrate that our algorithm outperforms the benchmarks under various scenarios.

Machine Learning,Networking and Internet Architecture

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily addresses the issue of optimizing information freshness in Mobile Edge Computing (MEC) systems. Specifically:

1. **Definition of Information Freshness**:
- The traditional definition of Age of Information (AoI) assumes that state information can be actively sampled and used directly. However, in many MEC applications, the required state information is updated in an event-driven manner and requires data processing to extract useful information. Therefore, the authors propose a new definition of AoI that considers event-driven data update strategies and additional data processing time.

2. **Online AoI Minimization Problem**:
- Based on the new AoI definition, the authors propose an online AoI minimization problem. The objective of this problem is to minimize the average age of information for all Wireless Devices (WDs) under limited spectrum bandwidth and energy consumption constraints.

3. **Application of Reinforcement Learning Algorithms**:
- Since traditional Reinforcement Learning (RL) algorithms converge slowly in completely unknown system dynamics, the authors introduce Post-Decision States (PDSs) to utilize partially known system dynamics, thereby accelerating the learning process. Additionally, by combining Deep Reinforcement Learning (DRL), particularly Deep Deterministic Policy Gradient (DDPG), the algorithm's applicability, scalability, and robustness are further improved.

Through these methods, the authors propose an efficient scheduling algorithm and validate its superior performance through extensive experiments.