Abstract:In this paper, we study the design of a distributive delay-optimal cross-layer scheduling algorithm for two-hop relay communication systems over frequency selective fading channels. The complex interactions of the queues at the source node and the M relays (RSs) are modeled as an infinite horizon average reward Markov Decision Process (MDP), whose state space involves the joint queue state (QSI) of the queue at the source node and the queues at the M RSs as well as the joint channel state (CSI) of all S-R links and R-D links. As a first step to address the curse of dimensionality, we propose a reduced state MDP formulation. From the associated Bellman’s equation, we show that the delay-optimal power control (and link selection algorithm), which are functions of both the CSI and QSI, has a multi-level water-filling structure. Furthermore, using stochastic learning, we derive a distributive online learning algorithm in which each node recursively estimates a per-node potential function based on real-time observations of the local CSI and local QSI only. Based on the real-time local potential estimates and using approximate MDP, we propose an auction-based algorithm for link selection and show that the combined distributive learning converges almost surely to a global optimal solution for large arrivals. The proposed online learning algorithm is different from the conventional online learning algorithms in two ways: (1) our online iterative solution updates both the value function (potential) and the Lagrange multipliers (LM) simultaneously; and (2) we establish the technical conditions for the September 10, 2009 DRAFT September 10, 2009 2 almost sure convergence even the per-node potential update equation is no longer a contraction mapping and the existing convergence results (based on contraction mapping) cannot be applied directly to our distributive stochastic learning algorithm. Finally, we show by simulation that the delay performance of the proposed scheme is significantly better than various baselines such as the conventional CSIT-only control and the throughput optimal control (in stability sense).

Novel Deep Reinforcement Learning‐based Delay‐constrained Buffer‐aided Relay Selection in Cognitive Cooperative Networks

Performance Analysis of the Cognitive Cooperative Scheme Based on Cognitive Relays

Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications

Optimal Relay Selection Strategy in Cognitive Cooperative Relaying Network

Distributed Relay Selection and Channel Choice in Cognitive Radio Network

Effective Relay Selection for Underwater Cooperative Acoustic Networks

Deep Reinforcement Learning-Based Relay Selection Algorithm in Free-Space Optical Cooperative Communications

Deep Q-Learning-Based Buffer-Aided Relay Selection for Reliable and Secure Communications in Two-Hop Wireless Relay Networks

Buffer-State-Based Probabilistic Relay Selection for Cooperative Networks With Delay Constraints

Deep Deterministic Policy Gradient for Relay Selection and Power Allocation in Cooperative Communication Network

Multi-Copy Relay Node Selection Strategy Based on Reinforcement Learning

Combined Relay Selection Enabled by Supervised Machine Learning

Delay-Optimal Two-Hop Cooperative Relay Communications via Approximate MDP and Distributive Stochastic Learning

A New Deep-Q-Learning-Based Transmission Scheduling Mechanism for the Cognitive Internet of Things

Partial Relay Selection with Fixed-Gain Relays and Outdated CSI in Underlay Cognitive Networks

Performance of Adaptive Link Selection With Buffer-Aided Relays in Underlay Cognitive Networks

Cooperative Relay to Improve Diversity in Cognitive Radio Networks

Energy-Efficient Distributed Relay and Power Control in Cognitive Radio Cooperative Communications.

Delay-Aware Two-Hop Cooperative Relay Communications Via Approximate MDP and Stochastic Learning

Joint Relay Selection and Beam Management Based on Deep Reinforcement Learning for Millimeter Wave Vehicular Communication

A Simple Cooperative Diversity Method Based on Deep-Learning-Aided Relay Selection