Abstract:A two-time-scale approximate Markov decision process (MDP) is proposed to optimize the uplink queuing performance of a millimeter wave (mmWave) communication system. Exploiting the wireless sensing techniques, the locations of signal reflectors, blockers, and mobile agents, as well as the mobility pattern of agents, become available knowledge to facilitate predictive scheduling. Notice that the state variation of the wireless channel fading and transmission queue is much faster than that of mobile agents’ locations. The joint optimization of uplink power adaptation, time allocation, and analog beamforming of all the frames is formulated as a two-time-scale MDP with the average uplink energy, queuing length, and buffer overflow rate in the minimization objective. The scheduling in the larger time scale with an infinite horizon, i.e., the analog transceiver beamforming, adapts with the random motion of mobile agents; whereas the scheduling in the smaller time scale with a finite horizon, i.e., uplink power and time allocation, adapts with both the small-scale channel fading, queue dynamics and large-scale random motion. The optimal solution of two-time-scale MDP requests iterative optimization between Bellman’s equations of both time scales, whose computation complexity is prohibitive. A novel low-complexity solution framework is then proposed to obtain the optimal larger-time-scale and sub-optimal smaller-time-scale policies, where the performance of the proposed scheme is bounded analytically. Benefiting from the motion sensing and blockage prediction, the proposed scheduling scheme outperforms existing benchmarks in the numerical simulations.

Delay-Aware Two-Hop Cooperative Relay Communications Via Approximate MDP and Stochastic Learning

Queue-Aware Distributive Resource Control for Delay-Sensitive Two-Hop MIMO Cooperative Systems

Relay Station Placement in Ieee 802.16j Dual-Relay Mmr Networks

Delay-Aware Massive Random Access for Machine-Type Communications Via Hierarchical Stochastic Learning

Generalized two-hop relay for flexible delay control in MANETs

Distributive Stochastic Learning for Delay-Optimal OFDMA Power and Subband Allocation

Stochastic Optimization for Joint Resource Allocation in OFDMA-Based Relay System

Partial Channel State Information Based Cooperative Relaying And Partner Selection

Dynamic Partial Cooperative MIMO System for Delay-Sensitive Applications with Limited Backhaul Capacity

Training Slot Allocation for Mitigating Estimation Error Propagation in A Two-Hop Relaying System

Delay-Aware Online Service Scheduling in High-Speed Railway Communication Systems

Cooperative Jamming in a Two-Hop Relay Wireless Network with Buffer-Aided Relays

Delay-Optimal User Scheduling and Inter-Cell Interference Management in Cellular Network via Distributive Stochastic Learning

Distributed joint optimization of relay selection and subchannel pairing in OFDM based relay networks

Novel Deep Reinforcement Learning‐based Delay‐constrained Buffer‐aided Relay Selection in Cognitive Cooperative Networks

Throughput-efficient Online Relay Selection for Dual-hop Cooperative Networks.

Balancing Performance and Cost for Two-Hop Cooperative Communications: Stackelberg Game and Distributed Multi-Agent Reinforcement Learning

Delay-Aware Two-Time-Scale Scheduling for mmWave Systems with Mobility and Environment Knowledge

Delay Optimal Scheduling for Cognitive Radios with Cooperative Beamforming: A Structured Matrix-Geometric Method

Buffer-State-Based Probabilistic Relay Selection for Cooperative Networks With Delay Constraints

Delay Optimal Scheduling for Cognitive Radio Networks with Cooperative Beamforming