An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems

Marwan Mousa,Damien van de Berg,Niki Kotecha,Ehecatl Antonio del Rio-Chanona,Max Mowbray

2023-07-21

Abstract:Most solutions to the inventory management problem assume a centralization of information that is incompatible with organisational constraints in real supply chain networks. The inventory management problem is a well-known planning problem in operations research, concerned with finding the optimal re-order policy for nodes in a supply chain. While many centralized solutions to the problem exist, they are not applicable to real-world supply chains made up of independent entities. The problem can however be naturally decomposed into sub-problems, each associated with an independent entity, turning it into a multi-agent system. Therefore, a decentralized data-driven solution to inventory management problems using multi-agent reinforcement learning is proposed where each entity is controlled by an agent. Three multi-agent variations of the proximal policy optimization algorithm are investigated through simulations of different supply chain networks and levels of uncertainty. The centralized training decentralized execution framework is deployed, which relies on offline centralization during simulation-based policy identification, but enables decentralization when the policies are deployed online to the real system. Results show that using multi-agent proximal policy optimization with a centralized critic leads to performance very close to that of a centralized data-driven solution and outperforms a distributed model-based solution in most cases while respecting the information constraints of the system.

Machine Learning,Multiagent Systems,Systems and Control

What problem does this paper attempt to address?

The paper primarily focuses on addressing the problem of Multi-Agent Reinforcement Learning (MARL) in decentralized inventory control systems. Specifically, the research aims to develop a decentralized, data-driven approach suitable for real-world supply chain networks to solve inventory management issues. Traditionally, most inventory management solutions assume centralized information, but this is incompatible with the actual organizational constraints of supply chain networks in reality. Therefore, the paper proposes a decentralized approach based on multi-agent reinforcement learning, where each independent entity is controlled by an agent. This approach allows each entity to make decisions independently based on local state information, thereby achieving true decentralized execution. The study examines three different versions of the Proximal Policy Optimization (PPO) algorithm, which are tested under simulations of different supply chain networks and levels of uncertainty. The adopted framework is Centralized Training Decentralized Execution (CTDE), meaning centralized training is conducted in offline simulations to identify optimal policies, while fully decentralized decision-making is implemented during online deployment. Experimental results indicate that the Multi-Agent Proximal Policy Optimization (MAPPO) method with a centralized critic can achieve performance close to that of centralized data-driven solutions and, in most cases, outperform distributed model-driven solutions while meeting the system's informational constraints. This demonstrates that the proposed method can effectively address the uncertainty issues in supply chain management while retaining decentralized characteristics.

An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems

Cooperative Multi-Agent Reinforcement Learning for Inventory Management

Reinforcement Learning for Multi-Product Multi-Node Inventory Management in Supply Chains

Multi-Agent Deep Reinforcement Learning for Multi-Echelon Inventory Management

Leveraging Graph Neural Networks and Multi-Agent Reinforcement Learning for Inventory Control in Supply Chains

A Deep Reinforcement Learning Approach for Inventory Control under Stochastic Lead Time and Demand

SCM network with multi-agent reinforcement learning

Multi-Agent Reinforcement Learning with Shared Resources for Inventory Management

Multi-Agent Reinforcement Learning with Shared Resources for Inventory Management

Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system

Dynamic inventory replenishment strategy for aerospace manufacturing supply chain: combining reinforcement learning and multi-agent simulation

MARLIM: Multi-Agent Reinforcement Learning for Inventory Management

Multi-echelon inventory optimization using deep reinforcement learning

Application of Multi-Agent Reinforcement Learning to Supply Chain Ordering Management

Multi-Agent Deep Reinforcement Learning for Recharging-Considered Vehicle Scheduling Problem in Container Terminals

A versatile multi-agent reinforcement learning benchmark for inventory management

Algorithmic Approaches to Inventory Management Optimization

Performance of deep reinforcement learning algorithms in two-echelon inventory control systems

A Multi-Agent Coordination of a Supply Chain Ordering Management with Multiple Members Using Reinforcement Learning