Abstract:Stochastic gradient descent (SGD) is a powerful optimization technique that is particularly useful in online learning scenarios. Its convergence analysis is relatively well understood under the assumption that the data samples are independent and identically distributed (iid). However, applying SGD to policy optimization problems in operations research involves a distinct challenge: the policy changes the environment and thereby affects the data used to update the policy. The adaptively generated data stream involves samples that are non-stationary, no longer independent from each other, and affected by previous decisions. The influence of previous decisions on the data generated introduces bias in the gradient estimate, which presents a potential source of instability for online learning not present in the iid case. In this paper, we introduce simple criteria for the adaptively generated data stream to guarantee the convergence of SGD. We show that the convergence speed of SGD with adaptive data is largely similar to the classical iid setting, as long as the mixing time of the policy-induced dynamics is factored in. Our Lyapunov-function analysis allows one to translate existing stability analysis of stochastic systems studied in operations research into convergence rates for SGD, and we demonstrate this for queueing and inventory management problems. We also showcase how our result can be applied to study the sample complexity of an actor-critic policy gradient algorithm.

Stochastic Gradient Descent in the Optimal Control of Execution Costs

Optimizing Execution Cost Using Stochastic Control

Stochastic Control for Optimal Execution: Fast Approximation Solution Scheme under Nested Mean-semi Deviation and Conditional Value at Risk

Optimal execution of limit and market orders with trade director, speed limiter, and fill uncertainty

On Parametric Optimal Execution and Machine Learning Surrogates

Policy gradient learning methods for stochastic control with exit time and applications to share repurchase pricing

Static vs Adaptive Strategies for Optimal Execution with Signals

An Optimal Execution Problem with a Geometric Ornstein-Uhlenbeck Price Process

Multi-asset optimal execution and statistical arbitrage strategies under Ornstein-Uhlenbeck dynamics

Deep Reinforcement Learning for Online Optimal Execution Strategies

Càdlàg semimartingale strategies for optimal trade execution in stochastic order book models

Optimal Execution Considering Trading Signal and Execution Risk Simultaneously

SDEs for Minimax Optimization

Optimal Trade Execution in an Order Book Model with Stochastic Liquidity Parameters

Distributed Stochastic Optimization with Random Communication and Computational Delays: Optimal Policies and Performance Analysis

Optimal execution with nonlinear transient market impact

Some Computations for Optimal Execution with Monotone Strategies

Optimal Execution with Reinforcement Learning

Optimal execution with dynamic risk adjustment

Stochastic Gradient Descent with Biased but Consistent Gradient Estimators

Stochastic Gradient Descent with Adaptive Data