MARCO: A Memory-Augmented Reinforcement Framework for Combinatorial Optimization

Andoni I. Garmendia,Quentin Cappart,Josu Ceberio,Alexander Mendiburu

DOI: https://doi.org/10.24963/ijcai.2024/766

2024-08-05

Abstract:Neural Combinatorial Optimization (NCO) is an emerging domain where deep learning techniques are employed to address combinatorial optimization problems as a standalone solver. Despite their potential, existing NCO methods often suffer from inefficient search space exploration, frequently leading to local optima entrapment or redundant exploration of previously visited states. This paper introduces a versatile framework, referred to as Memory-Augmented Reinforcement for Combinatorial Optimization (MARCO), that can be used to enhance both constructive and improvement methods in NCO through an innovative memory module. MARCO stores data collected throughout the optimization trajectory and retrieves contextually relevant information at each state. This way, the search is guided by two competing criteria: making the best decision in terms of the quality of the solution and avoiding revisiting already explored solutions. This approach promotes a more efficient use of the available optimization budget. Moreover, thanks to the parallel nature of NCO models, several search threads can run simultaneously, all sharing the same memory module, enabling an efficient collaborative exploration. Empirical evaluations, carried out on the maximum cut, maximum independent set and travelling salesman problems, reveal that the memory module effectively increases the exploration, enabling the model to discover diverse, higher-quality solutions. MARCO achieves good performance in a low computational cost, establishing a promising new direction in the field of NCO.

Neural and Evolutionary Computing,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address a problem in the field of Combinatorial Optimization (CO), specifically how to improve Neural Combinatorial Optimization (NCO) methods by introducing a memory-augmented mechanism. Specifically:

1. **Limitations of Existing NCO Methods**: Existing NCO methods are inefficient in exploring the search space, often getting stuck in local optima or repeatedly exploring already visited states.
2. **Proposing the MARCO Framework**: The paper introduces a new framework called "Memory-Augmented Reinforcement for Combinatorial Optimization" (MARCO), which enhances the construction and improvement methods in NCO through an innovative memory module. MARCO can store data collected throughout the optimization process and retrieve relevant historical information at each state, guiding the search process to avoid repeatedly exploring already visited solutions.
3. **Improving Exploration Efficiency**: MARCO guides decision-making by recording historical data, avoiding repeated exploration of the same states, thereby improving exploration efficiency. Additionally, MARCO leverages the parallel nature of NCO models, allowing multiple search threads to share the same memory module, achieving efficient collaborative exploration.

The paper empirically evaluates MARCO on the Maximum Cut, Maximum Independent Set, and Travelling Salesman Problem, demonstrating the effectiveness and superiority of MARCO. Experimental results show that MARCO can achieve good performance at a lower computational cost, providing new research directions for the NCO field.