Abstract:We argue that inventory management presents unique opportunities for reliably applying and evaluating deep reinforcement learning (DRL). Toward reliable application, we emphasize and test two techniques. The first is Hindsight Differentiable Policy Optimization (HDPO), which performs stochastic gradient descent to optimize policy performance while avoiding the need to repeatedly deploy randomized policies in the environment-as is common with generic policy gradient methods. Our second technique involves aligning policy (neural) network architectures with the structure of the inventory network. Specifically, we focus on a network with a single warehouse that consolidates inventory from external suppliers, holds it, and then distributes it to many stores as needed. In this setting, we introduce the symmetry-aware policy network architecture. We motivate this architecture by establishing an asymptotic performance guarantee and empirically demonstrate its ability to reduce the amount of data needed to uncover strong policies. Both techniques exploit structures inherent in inventory management problems, moving beyond generic DRL algorithms. Toward rigorous evaluation, we create and share new benchmark problems, divided into two categories. One type focuses on problems with hidden structures that allow us to compute or bound the cost of the true optimal policy. Across four problems of this type, we find HDPO consistently attains near-optimal performance, handling up to 60-dimensional raw state vectors effectively. The other type of evaluation involves constructing a test problem using real time series data from a large retailer, where the optimum is poorly understood. Here, we find HDPO methods meaningfully outperform a variety of generalized newsvendor heuristics. Our code can be found at <a class="link-external link-http" href="http://github.com/MatiasAlvo/Neural_inventory_control" rel="external noopener nofollow">this http URL</a>.

Newsvendor Model with Deep Reinforcement Learning

Applying Deep Learning to the Newsvendor Problem

Deep Inventory Management

Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization

Dual-Agent Deep Reinforcement Learning for Dynamic Pricing and Replenishment

A Deep Reinforcement Learning Approach for Inventory Control under Stochastic Lead Time and Demand

Deep Generative Demand Learning for Newsvendor and Pricing

Deep reinforcement learning for demand fulfillment in online retail

From Bandits Model to Deep Deterministic Policy Gradient, Reinforcement Learning with Contextual Information

Deep differentiable reinforcement learning and optimal trading

Deep Reinforcement Learning for inventory optimization with non-stationary uncertain demand

An end-to-end deep learning model for solving data-driven newsvendor problem with accessibility to textual review data

Deep Robust Reinforcement Learning for Practical Algorithmic Trading

Deep Reinforcement Trading with Predictable Returns

Neural Inventory Control in Networks via Hindsight Differentiable Policy Optimization

Deep Neural Newsvendor

Solving Inventory Management Problems Through Deep Reinforcement Learning

Deep Reinforcement Learning Agents for Strategic Production Policies in Microeconomic Market Simulations

Model-based Reinforcement Learning for Predictions and Control for Limit Order Books

Reinforcement Learning in Non-Markov Market-Making

Deep Deterministic Portfolio Optimization