Reinforcement Learning with Intrinsically Motivated Feedback Graph for Lost-sales Inventory Control

Zifan Liu,Xinran Li,Shibo Chen,Gen Li,Jiashuo Jiang,Jun Zhang

2024-06-26

Abstract:Reinforcement learning (RL) has proven to be well-performed and general-purpose in the inventory control (IC). However, further improvement of RL algorithms in the IC domain is impeded due to two limitations of online experience. First, online experience is expensive to acquire in real-world applications. With the low sample efficiency nature of RL algorithms, it would take extensive time to train the RL policy to convergence. Second, online experience may not reflect the true demand due to the lost sales phenomenon typical in IC, which makes the learning process more challenging. To address the above challenges, we propose a decision framework that combines reinforcement learning with feedback graph (RLFG) and intrinsically motivated exploration (IME) to boost sample efficiency. In particular, we first take advantage of the inherent properties of lost-sales IC problems and design the feedback graph (FG) specially for lost-sales IC problems to generate abundant side experiences aid RL updates. Then we conduct a rigorous theoretical analysis of how the designed FG reduces the sample complexity of RL methods. Based on the theoretical insights, we design an intrinsic reward to direct the RL agent to explore to the state-action space with more side experiences, further exploiting FG's power. Experimental results demonstrate that our method greatly improves the sample efficiency of applying RL in IC. Our code is available at https://anonymous.4open.science/r/RLIMFG4IC-811D/

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

### The Problem Addressed by This Paper This paper aims to address the issue of sample efficiency in Inventory Control (IC), particularly for the "lost-sales" type of inventory control problem. Specifically: 1. **High Cost of Online Experience Acquisition**: In real-world applications, acquiring online experience is very expensive and time-consuming. Reinforcement learning algorithms typically have low sample efficiency, making the training process very slow. 2. **Demand Obscuration**: The "lost-sales" phenomenon leads to the true demand being unobservable, further increasing the difficulty of learning. To tackle these issues, the authors propose a decision-making framework that combines Feedback Graph (FG) and Intrinsically Motivated Exploration (IME), referred to as RLFG. This method improves sample efficiency by generating a large amount of side experiences and designs an intrinsic reward mechanism to guide the reinforcement learning agent to better explore the state-action space, thereby further enhancing learning efficiency. Experimental results show that this method significantly improves sample efficiency in standard lost-sales inventory control environments.

Reinforcement Learning with Intrinsically Motivated Feedback Graph for Lost-sales Inventory Control

Deep Controlled Learning for Inventory Control

Solving Inventory Management Problems Through Deep Reinforcement Learning

Can Deep Reinforcement Learning Improve Inventory Management? Performance on Dual Sourcing, Lost Sales and Multi-Echelon Problems

A Reinforcement Learning Method for Inventory Control under State-based Stochastic Demand

Deep Policy Iteration with Integer Programming for Inventory Management

RLIF: Interactive Imitation Learning as Reinforcement Learning

Cooperative Multi-Agent Reinforcement Learning for Inventory Management

Deep Inventory Management

Reward shaping to improve the performance of deep reinforcement learning in perishable inventory management

A Deep Reinforcement Learning Approach for Inventory Control under Stochastic Lead Time and Demand

Intrinsically Motivated Reinforcement Learning Based Recommendation with Counterfactual Data Augmentation

Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system

Deep Reinforcement Learning for Large-Scale Inventory Management

Performance of deep reinforcement learning algorithms in two-echelon inventory control systems

Neural Inventory Control in Networks via Hindsight Differentiable Policy Optimization

RIRL: A Recurrent Imitation and Reinforcement Learning Method for Long-Horizon Robotic Tasks

Deep Reinforcement Learning for inventory optimization with non-stationary uncertain demand

Constrained continuous-action reinforcement learning for supply chain inventory management

Blending Imitation and Reinforcement Learning for Robust Policy Improvement

Deep Reinforcement Learning Approach for Solving Joint Pricing and Inventory Problem with Reference Price Effects