Abstract:Combinatorial optimization is the field devoted to the study and practice of algorithms that solve NP-hard problems. As Machine Learning (ML) and deep learning have popularized, several research groups have started to use ML to solve combinatorial optimization problems, such as the well-known Travelling Salesman Problem (TSP). Based on deep (reinforcement) learning, new models and architecture for the TSP have been successively developed and have gained increasing performances. At the time of writing, state-of-the-art models provide solutions to TSP instances of 100 cities that are roughly 1.33% away from optimal solutions. However, despite these apparently positive results, the performances remain far from those that can be achieved using a specialized search procedure. In this paper, we address the limitations of ML approaches for solving the TSP and investigate two fundamental questions: (1) how can we measure the level of accuracy of the pure ML component of such methods; and (2) what is the impact of a search procedure plugged inside a ML model on the performances? To answer these questions, we propose a new metric, ratio of optimal decisions (ROD), based on a fair comparison with a parametrized oracle, mimicking a ML model with a controlled accuracy. All the experiments are carried out on four state-of-the-art ML approaches dedicated to solve the TSP. Finally, we made ROD open-source in order to ease future research in the field.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to evaluate the performance of machine learning (ML) methods in combinatorial optimization problems, especially for the Traveling Salesman Problem (TSP). Although new models and architectures based on deep (reinforcement) learning have made significant progress in solving TSP, their performance is still far inferior to that of specialized search programs. Therefore, this paper raises two fundamental questions: 1. How to measure the accuracy of the pure machine - learning components in these methods? 2. What is the impact of the search program on the performance embedded in the machine - learning model? To answer these questions, the authors propose a new metric - the Ratio of Optimal Decisions (ROD), which is based on a fair comparison with a parameterized oracle that can simulate a machine - learning model with controllable accuracy. ROD aims to evaluate the accuracy of the learning component independently of the search component. In this way, researchers can more clearly understand the relative importance of learning and searching in machine - learning - based combinatorial optimization methods. ### Main contributions of the paper 1. **Proposing the ROD metric**: This is a new evaluation index for evaluating the accuracy of the learning part when machine - learning methods solve combinatorial optimization problems. 2. **Re - evaluating existing models**: The ROD metric is applied to re - evaluate the performance of four state - of - the - art machine - learning models in solving the TSP problem. 3. **Empirical analysis**: Empirical evidence is provided, indicating that the design of the search program has a huge impact on the overall performance of machine - learning methods. 4. **Open - sourcing ROD**: The ROD metric is open - sourced to promote future research in the field of combinatorial optimization. ### Experimental setup - **Data set**: Two test sets are used, each containing 1,000 randomly generated 2D Euclidean graphs with 50 and 100 vertices respectively. - **Model selection**: Four different machine - learning models are selected, which differ in neural network architecture and learning algorithm. - **Evaluation method**: The ROD metric is implemented, and the Concorde solver is used to calculate the optimal paths of instances. ### Results - **Application of the ROD metric**: The results show that even the best machine - learning model is far inferior to traditional operations research methods in terms of optimality gap, but the performance of its learning component is comparable to that of a high - precision oracle. - **Impact of the search program**: Experiments prove that the design of the search program has a significant impact on the performance of machine - learning methods. ### Conclusion By introducing the ROD metric, this paper provides a new perspective for evaluating the performance of machine - learning methods in combinatorial optimization problems. The ROD metric not only helps to understand the roles of learning and searching in these methods, but also provides a valuable tool for future research.

How to Evaluate Machine Learning Approaches for Combinatorial Optimization: Application to the Travelling Salesman Problem

Position: Rethinking Post-Hoc Search-Based Neural Approaches for Solving Large-Scale Traveling Salesman Problems

Local search for the Traveling Salesman Problem: A comparative study

A New Constructive Heuristic Driven by Machine Learning for the Traveling Salesman Problem

A Novel Feature-Based Approach to Characterize Algorithm Performance for the Traveling Salesman Problem

How Good is Neural Combinatorial Optimization? A Systematic Evaluation on the Traveling Salesman Problem

Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization

The Transformer Network for the Traveling Salesman Problem

Leveraging TSP Solver Complementarity through Machine Learning

A novel stochastic method with modified extremal optimization and nearest neighbor search for hard combinatorial problems

Improvements for mlrose applied to the Traveling Salesperson Problem

Machine Learning Constructives and Local Searches for the Travelling Salesman Problem

Neural Combinatorial Optimization with Reinforcement Learning

Benchmarking Optimization Algorithms: an Open Source Framework for the Traveling Salesman Problem

Combining Reinforcement Learning and Optimal Transport for the Traveling Salesman Problem

The Machine Learning for Combinatorial Optimization Competition (ML4CO): Results and Insights.

Decomposition-based Lin-Kernighan Heuristic with Neighborhood Structure Transfer for Multi/Many-objective Traveling Salesman Problem

Deep reinforcement learning for multi-objective combinatorial optimization: A case study on multi-objective traveling salesman problem

A Graph Pointer Network-Based Multi-Objective Deep Reinforcement Learning Algorithm for Solving the Traveling Salesman Problem

A deep reinforcement learning algorithm framework for solving multi-objective traveling salesman problem based on feature transformation

A Survey for Solving Mixed Integer Programming via Machine Learning