Abstract:We study the problem of zero-delay coding for the transmission a Markov source over a noisy channel with feedback and present a rigorous reinforcement theoretic solution which is guaranteed to achieve near-optimality. To this end, we formulate the problem as a Markov decision process (MDP) where the state is a probability-measure valued predictor/belief and the actions are quantizer maps. This MDP formulation has been used to show the optimality of certain classes of encoder policies in prior work. Despite such an analytical approach in determining optimal policies, their computation is prohibitively complex due to the uncountable nature of the constructed state space and the lack of minorization or strong ergodicity results which are commonly assumed for average cost optimal stochastic control. These challenges invite rigorous reinforcement learning methods, which entail several open questions addressed in our paper. We present two complementary approaches for this problem. In the first approach, we approximate the set of all beliefs by a finite set and use nearest-neighbor quantization to obtain a finite state MDP, whose optimal policies become near-optimal for the original MDP as the quantization becomes arbitrarily fine. In the second approach, a sliding finite window of channel outputs and quantizers together with a prior belief state serve as the state of the MDP. We then approximate this state by marginalizing over all possible beliefs, so that our coding policies only use the finite window term to encode the source. Under an appropriate notion of predictor stability, we show that such policies are near-optimal for the zero-delay coding problem as the window length increases. We give sufficient conditions for predictor stability to hold. Finally, we propose a reinforcement learning algorithm to compute near-optimal policies.

Reinforcement Learning for Jointly Optimal Coding and Control over a Communication Channel

Reinforcement Learning for Optimal Transmission of Markov Sources over Noisy Channels: Belief Quantization vs Sliding Finite Window Codes

Approximate Optimal Control over Unreliable Communication Channels

Reinforcement Learning for Near-Optimal Design of Zero-Delay Codes for Markov Sources

Online Optimization for Learning to Communicate over Time-Correlated Channels

Reinforcement Learning Control for Discrete-time Systems over Fading Channels

Inverse optimal stabilization of cooperative control in networked multi-agent systems

Control-theoretic Approach to Communication with Feedback: Fundamental Limits and Code Design

Quantized iterative learning control of communication-constrained systems with encoding and decoding mechanism

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Dynamic Programming-based Approximate Optimal Control for Model-Based Reinforcement Learning

Learning in Wireless Control Systems over Non-Stationary Channels

Scalable Reinforcement Learning for Linear-Quadratic Control of Networks

Information-Theoretic Viewpoints on Optimal Causal Coding-Decoding Problems

Causality-Driven Reinforcement Learning for Joint Communication and Sensing

Robust Non-Linear Feedback Coding via Power-Constrained Deep Learning

A Reinforcement Learning Approach to Jointly Adapt Vehicular Communications and Planning for Optimized Driving

Networked Control with Hybrid Automatic Repeat Request Protocols

Reinforcement Learning for Channel Coding: Learned Bit-Flipping Decoding

Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation

Data-Driven Networked Optimal Iterative Learning Control for Discrete Linear Time-Varying Systems with One-Operation Bernoulli-Type Communication Delays