Off-line approximate dynamic programming for the vehicle routing problem with a highly variable customer basis and stochastic demands

Mohsen Dastpak,Fausto Errico,Ola Jabali

DOI: https://doi.org/10.48550/arXiv.2109.10200

2022-07-14

Abstract:We study a stochastic variant of the vehicle routing problem arising in the context of domestic donor collection services. The problem we consider combines the following attributes. Customers requesting services are variable, in the sense that the customers are stochastic but are not restricted to a predefined set, as they may appear anywhere in a given service area. Furthermore, demand volumes are stochastic and observed upon visiting the customer. The objective is to maximize the expected served demands while meeting vehicle capacity and time restrictions. We call this problem the VRP with a highly Variable Customer basis and Stochastic Demands (VRP-VCSD). For this problem, we first propose a Markov Decision Process (MDP) formulation representing the classical centralized decision-making perspective where one decision-maker establishes the routes of all vehicles. While the resulting formulation turns out to be intractable, it provides us with the ground to develop a new MDP formulation, which we call partially decentralized. In this formulation, the action-space is decomposed by vehicle. However, the decentralization is incomplete as we enforce identical vehicle-specific policies while optimizing the collective reward. We propose several strategies to reduce the dimension of the state and action spaces associated with the partially decentralized formulation. These yield a considerably more tractable problem, which we solve via Reinforcement Learning. In particular, we develop a Q-learning algorithm called DecQN, featuring state-of-the-art acceleration techniques. We conduct a thorough computational analysis. Results show that DecQN considerably outperforms three benchmark policies. Moreover, we show that our approach can compete with specialized methods developed for the particular case of the VRP-VCSD, where customer locations and expected demands are known in advance.

Optimization and Control,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the Vehicle Routing Problem with a Highly Variable Customer Basis and Stochastic Demands (VRP - VCSD) in the context of Domestic Donor Collection Services (DDCSs). Specifically, the paper focuses on:

1. **High variability of the customer base**: Unlike traditional vehicle routing problems, the customers here are not a fixed set but randomly emerge from a larger pool of potential customers, which increases the complexity of the problem.
2. **Randomness of demands**: Not only is the appearance of customers random, but the specific demand of each customer can only be determined after the service vehicle arrives, which further increases the uncertainty of the problem.
3. **Objective function**: Maximize the total expected service demand under the premise of meeting vehicle capacity and time constraints.

To solve these problems, the author proposes the following methods:

- **Markov Decision Process (MDP) modeling**: First, the author uses the classical centralized decision - making framework to model VRP - VCSD with MDP. However, due to the multi - vehicle nature of the problem, the dimensions of the state and action spaces expand dramatically, making this model difficult to solve.
- **Partially decentralized MDP model**: To reduce the dimensions of the state and action spaces, the author proposes a partially decentralized MDP model. In this model, although the action space of each vehicle is considered separately, all vehicles share the same policy and can access the state information of the entire system when making decisions.
- **Reinforcement learning algorithm (Q - learning)**: The author develops a Q - learning algorithm called DecQN, which combines the latest acceleration techniques, such as Replay Memory and Double Q Network, to improve algorithm performance. Through simulation experiments, DecQN has shown significant advantages in multiple benchmark tests.

Overall, the main contribution of this paper lies in introducing a new VRP variant - VRP - VCSD and proposing an effective solution that can optimize vehicle routing planning while dealing with a highly variable customer base and random demands.