Abstract:Combinatorial optimization has found applications in numerous fields, from aerospace to transportation planning and economics. The goal is to find an optimal solution among a finite set of possibilities. The well-known challenge one faces with combinatorial optimization is the state-space explosion problem: the number of possibilities grows exponentially with the problem size, which makes solving intractable for large problems. In the last years, deep reinforcement learning (DRL) has shown its promise for designing good heuristics dedicated to solve NP-hard combinatorial optimization problems. However, current approaches have two shortcomings: (1) they mainly focus on the standard travelling salesman problem and they cannot be easily extended to other problems, and (2) they only provide an approximate solution with no systematic ways to improve it or to prove optimality. In another context, constraint programming (CP) is a generic tool to solve combinatorial optimization problems. Based on a complete search procedure, it will always find the optimal solution if we allow an execution time large enough. A critical design choice, that makes CP non-trivial to use in practice, is the branching decision, directing how the search space is explored. In this work, we propose a general and hybrid approach, based on DRL and CP, for solving combinatorial optimization problems. The core of our approach is based on a dynamic programming formulation, that acts as a bridge between both techniques. We experimentally show that our solver is efficient to solve two challenging problems: the traveling salesman problem with time windows, and the 4-moments portfolio optimization problem. Results obtained show that the framework introduced outperforms the stand-alone RL and CP solutions, while being competitive with industrial solvers.

Stochastic Constraint Programming as Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Delay-Aware Stochastic Resource Management for Mobile Edge Computing Systems Via Constrained Reinforcement Learning

Stochastic Constraint Programming: A Scenario-Based Approach

Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization

Stochastic Constraint Programming

State-wise Constrained Policy Optimization

Leveraging Constraint Programming in a Deep Learning Approach for Dynamically Solving the Flexible Job-Shop Scheduling Problem

Neural-Progressive Hedging: Enforcing Constraints in Reinforcement Learning with Stochastic Programming

Proactive and Reactive Constraint Programming for Stochastic Project Scheduling with Maximal Time-Lags

Constrained Combinatorial Optimization with Reinforcement Learning

Semi-Infinitely Constrained Markov Decision Processes and Provably Efficient Reinforcement Learning.

Constrained Reinforcement Learning Under Model Mismatch

SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization

Probabilistic Constraint for Safety-Critical Reinforcement Learning

Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learning

Enhancing Constraint Programming via Supervised Learning for Job Shop Scheduling

Reinforcement Learning and Mixed-Integer Programming for Power Plant Scheduling in Low Carbon Systems: Comparison and Hybridisation

Resilient Constrained Reinforcement Learning

Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction