Abstract:We design a new iterative algorithm, called REINFORCE-OPT, for solving a general type of optimization problems. This algorithm parameterizes the solution-searching rule and iteratively improve the parameter using a reinforcement learning (RL) algorithm that resembles REINFORCE. To provide a deeper understanding of the RL-based methods, we prove that the goal of REINFORCE-OPT is equivalent to solving a stochastic-version of the given optimization problem, and that under standard assumptions, the searching-rule parameter almost surely converges to a locally optimal value. Experiments show that REINFORCE-OPT outperforms other optimization methods, such as gradient descent, genetic algorithm, and particle swarm optimization, by its ability to escape form locally optimal solutions and its robustness to the choice of initial values. Then, we formally introduce, with rigorous derivations, the use of reinforcement learning to the field of inverse problems. By choosing specific probability models for the action-selection rule, we connect our approach to the conventional regularization methods of Tikhonov regularization and iterative regularization. Our work provides two typical examples (non-linear integral equations and parameter-identification problems in partial differential equations) on how reinforcement learning can be applied in solving non-linear inverse problems. The numerical experiments highlight the strong performance of REINFORCE-OPT, along with its ability to quantify uncertainty in error estimates and identify multiple solutions for ill-posed inverse problems without stability and uniqueness.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper primarily attempts to address the following two problems: 1. **Optimization Problem**: - Proposes a new iterative algorithm, REINFORCE-OPT, to solve a class of generalized optimization problems. - Demonstrates that the objective of REINFORCE-OPT is equivalent to solving a stochastic version of the given optimization problem, and under standard assumptions, the search rule parameters almost surely converge to a local optimum. - Experiments show that REINFORCE-OPT has a stronger ability to escape local optima compared to other optimization methods (such as gradient descent, genetic algorithms, and particle swarm optimization) and is more robust to the choice of initial values. 2. **Application to Inverse Problems**: - Formally introduces the use of reinforcement learning to solve inverse problems and connects this method with traditional regularization methods (such as Tikhonov regularization and iterative regularization) by selecting specific probabilistic models. - Demonstrates how to apply reinforcement learning to solve nonlinear inverse problems through two typical examples: parameter identification problems in nonlinear integral equations and partial differential equations. - Numerical experiments show that REINFORCE-OPT not only performs strongly but also quantifies uncertainty in error estimation and identifies multiple solutions to ill-posed inverse problems. ### Main Contributions - **Theoretical Contribution**: Advances the understanding of using reinforcement learning methods to solve potential non-convex optimization problems in continuous solution spaces. Establishes for the first time the equivalence between complex objectives and stochastic versions of optimization objectives, and provides a theoretical convergence analysis of reinforcement learning optimization methods. - **Introduction to the Field of Inverse Problems**: Systematically introduces reinforcement learning algorithms to the field of inverse problems for the first time and combines them with classical regularization methods. - **Practical Application**: Numerical experiments demonstrate the practicality and success cases of REINFORCE-OPT in solving complexly structured inverse problems.

Solving Optimization Problems Using Reinforcement Learning, with Applications to Inverse Problems

Off-Dynamics Inverse Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Dynamic Multiobjective Optimization Driven by Inverse Reinforcement Learning.

Reinforcement Learning Driven Heuristic Optimization

A Population-Based Incremental Learning Method for Robust Optimal Solutions

Learning to Optimize for Reinforcement Learning

Optimization Learning: Perspective, Method, and Applications.

Hybrid Inverse Reinforcement Learning

A Convex Programming Approach to Data-Driven Risk-Averse Reinforcement Learning

Optimization Methods for Inverse Problems

Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

A Deep Reinforcement Learning Approach to Efficient Distributed Optimization

Solving Inverse Problems via Diffusion Optimal Control

Can Learned Optimization Make Reinforcement Learning Less Difficult?

Self-Supervised Learning of Iterative Solvers for Constrained Optimization

Solving the K-Sparse Eigenvalue Problem with Reinforcement Learning

Teaching Networks to Solve Optimization Problems

A novel reinforcement learning-based method for structure optimization

Hybrid Reinforcement Learning Framework for Mixed-Variable Problems