The existence of random solutions to random optimization problems

Ta Ngoc Anh

DOI: https://doi.org/10.48550/arXiv.2005.06233

2020-05-13

Abstract:In this paper we study random optimization problems where random functions are investigated in sample paths. Some sufficient conditions ensuring the existence of random solutions to random optimization problems are proposed.

Probability

What problem does this paper attempt to address?

A novel stochastic method with modified extremal optimization and nearest neighbor search for hard combinatorial problems

Guo-Qiang Zeng,Yong-Zai Lu,Wei-Jie Mao

DOI: https://doi.org/10.1109/WCICA.2010.5554748

2010-01-01

Intelligent Control and Automation

Abstract:The combinatorial optimization occurs in many real-world problems including the fields of engineering, physics and economics. It has been recognized that some problems with highly degenerate states are difficult to solve in terms of many existing optimization algorithms. This paper proposes a novel stochastic method with modified extremal optimization (EO) and nearest neighbor search to deal with these problems. It starts from making use of the recent discovered statistical property to generate the initial configurations by the nearest neighbor search and then further explores the complex configuration spaces by a modified EO approach that applies more general probability distributions-based evolution mechanism. The experimental results with some hard instances of traveling salesman problem (TSP), a popular benchmark for combinatorial optimization problems demonstrate that the proposed method may provide better performance than other physics-inspired algorithms such as simulated annealing, EO and self-organized algorithm.
On the stability of solutions to random optimization problems under small perturbations

Sourav Chatterjee,Souvik Ray

2024-10-29

Abstract:Consider the Euclidean traveling salesman problem with $n$ random points on the plane. Suppose that one of the points is shifted to a new random location. This gives us a new optimal path. Consider such shifts for each of the $n$ points. Do we get $n$ very different optimal paths? In this article, we show that this is not the case - in fact, the number of truly different paths can be at most $\mathcal{O}(1)$ as $n\to \infty$. The proof is based on a general argument which allows us to prove similar stability results in a number of other settings, such as branching random walk, the Sherrington-Kirkpatrick model of mean-field spin glasses, the Edwards-Anderson model of short-range spin glasses, and the Wigner ensemble of random matrices.

Probability,Discrete Mathematics,Mathematical Physics,Combinatorics
Studies on Some Modern Optimization Algorithms

徐宁,李春光,张健,虞厥邦

DOI: https://doi.org/10.3321/j.issn:1001-506X.2002.12.030

2002-01-01

Abstract:Modern optimization includes genetic algorithm, ant colony algorithm and tabu search algorithm. They are mainly applied to solve difficult optimization problems. They are widely used independent of gradient, so they are suitable to solve large scale complicated problems which are unsolvable in traditional methods. This paper first introduces the basic ideas of these algorithms, and then gives their characteristics and comparison. Finally, future research directions are pointed out.\;
A Bound on the Expected Optimality of Random Feasible Solutions to Combinatorial Optimization Problems

Evan A. Sultanik

DOI: https://doi.org/10.48550/arXiv.1402.0423

2014-02-04

Abstract:This paper investigates and bounds the expected solution quality of combinatorial optimization problems when feasible solutions are chosen at random. Loose general bounds are discovered, as well as families of combinatorial optimization problems for which random feasible solutions are expected to be a constant factor of optimal. One implication of this result is that, for graphical problems, if the average edge weight in a feasible solution is sufficiently small, then any randomly chosen feasible solution to the problem will be a constant factor of optimal. For example, under certain well-defined circumstances, the expected constant of approximation of a randomly chosen feasible solution to the Steiner network problem is bounded above by 3. Empirical analysis supports these bounds and actually suggest that they might be tightened.

Data Structures and Algorithms,Combinatorics
Algorithm-independent bounds on complex optimization through the statistics of marginal optima

Jaron Kent-Dobias

2024-07-24

Abstract:Optimization seeks extremal points in a function. When there are superextensively many optima, optimization algorithms are liable to get stuck. Under these conditions, generic algorithms tend to find marginal optima, which have many nearly flat directions. In a companion paper, we introduce a technique to count marginal optima in random landscapes. Here, we use the statistics of marginal optima calculated using this technique to produce generic bounds on optimization, based on the simple principle that algorithms will overwhelmingly tend to get stuck only where marginal optima are found. We demonstrate the idea on a simple non-Gaussian problem of maximizing the sum of squared random functions on a compact space. Numeric experiments using both gradient descent and generalized approximate message passing algorithms fall inside the expected bounds.

Disordered Systems and Neural Networks,Statistical Mechanics
Some Remarks on the Optimal Level of Randomization in Global Optimization

Ted Theodosopoulos

DOI: https://doi.org/10.48550/arXiv.math/0406095

2004-06-06

Abstract:For a class of stochastic restart algorithms we address the effect of a nonzero level of randomization in maximizing the convergence rate for general energy landscapes. The resulting characterization of the optimal level of randomization is investigated computationally for random as well as parametric families of rugged energy landscapes.

Optimization and Control,Numerical Analysis,Probability
Simulation-Based Discrete Optimization of Stochastic Discrete Event Systems Subject to Non Closed-Form Constraints.

Jie Li,Alexandre Sava,Xiaolan Xie

DOI: https://doi.org/10.1109/tac.2009.2033847

IF: 6.549

2009-01-01

IEEE Transactions on Automatic Control

Abstract:This technical note addresses the discrete optimization of stochastic discrete event systems for which both the performance function and the constraint function are not known but can be evaluated by simulation and the solution space is either finite or unbounded. Our method is based on random search in a neighborhood structure called the most promising area proposed in and a moving observation area. The simulation budget is allocated dynamically to promising solutions. Simulation-based constraints are taken into account in an augmented performance function via an increasing penalty factor. We prove that under some assumptions, the algorithm converges with probability 1 to a set of true local optimal solutions. These assumptions are restrictive and difficult to verify but we hope that the encouraging numerical results would motivate future research exploiting ideas of this technical note.
Existence theorems for optimal solutions in semi-algebraic optimization

Jae Hyoung Lee,Gue Myung Lee,Tien Son Pham

2023-08-10

Abstract:Consider the problem of minimizing a lower semi-continuous semi-algebraic function $f \colon \mathbb{R}^n \to \mathbb{R} \cup \{+\infty\}$ on an unbounded closed semi-algebraic set $S \subset \mathbb{R}^n.$ Employing adequate tools of semi-algebraic geometry, we first establish some properties of the tangency variety of the restriction of $f$ on $S.$ Then we derive verifiable necessary and sufficient conditions for the existence of optimal solutions of the problem as well as the boundedness from below and coercivity of the restriction of $f$ on $S.$ We also present a computable formula for the optimal value of the problem.

Optimization and Control
A Note On The Popularity of Stochastic Optimization Algorithms in Different Fields: A Quantitative Analysis from 2007 to 2017

Son Duy Dao

DOI: https://doi.org/10.48550/arXiv.1907.01453

2019-06-30

Neural and Evolutionary Computing

Abstract:Stochastic optimization algorithms are often used to solve complex large-scale optimization problems in various fields. To date, there have been a number of stochastic optimization algorithms such as Genetic Algorithm, Cuckoo Search, Tabu Search, Simulated Annealing, Particle Swarm Optimization, Ant Colony Optimization, etc. Each algorithm has some advantages and disadvantages. Currently, there is no study that can help researchers to choose the most popular optimization algorithm to deal with the problems in different research fields. In this note, a quantitative analysis of the popularity of 14 stochastic optimization algorithms in 18 different research fields in the last ten years from 2007 to 2017 is provided. This quantitative analysis can help researchers/practitioners select the best optimization algorithm to solve complex large-scale optimization problems in the fields of Engineering, Computer science, Operations research, Mathematics, Physics, Chemistry, Automation control systems, Materials science, Energy fuels, Mechanics, Telecommunications, Thermodynamics, Optics, Environmental sciences ecology, Water resources, Transportation, Construction building technology, and Robotics.
Hardness of Random Optimization Problems for Boolean Circuits, Low-Degree Polynomials, and Langevin Dynamics

David Gamarnik,Aukosh Jagannath,Alexander S. Wein

DOI: https://doi.org/10.1137/22m150263x

2024-02-14

SIAM Journal on Computing

Abstract:SIAM Journal on Computing, Volume 53, Issue 1, Page 1-46, February 2024. We consider the problem of finding nearly optimal solutions of optimization problems with random objective functions. Such problems arise widely in the theory of random graphs, theoretical computer science, and statistical physics. Two concrete problems we consider are (a) optimizing the Hamiltonian of a spherical or Ising [math]-spin glass model and (b) finding a large independent set in a sparse Erdős–Rényi graph. The following families of algorithms are considered: (a) low-degree polynomials of the input—a general framework that captures many prior algorithms; (b) low-depth Boolean circuits; (c) the Langevin dynamics algorithm, a canonical Monte Carlo analogue of the gradient descent algorithm. We show that these families of algorithms cannot have high success probability. For the case of Boolean circuits, our results improve the state-of-the-art bounds known in circuit complexity theory (although we consider the search problem as opposed to the decision problem). Our proof uses the fact that these models are known to exhibit a variant of the overlap gap property (OGP) of near-optimal solutions. Specifically, for both models, every two solutions whose objectives are above a certain threshold are either close to or far from each other. The crux of our proof is that the classes of algorithms we consider exhibit a form of stability (noise-insensitivity): a small perturbation of the input induces a small perturbation of the output. We show by an interpolation argument that stable algorithms cannot overcome the OGP barrier. The stability of Langevin dynamics is an immediate consequence of the well-posedness of stochastic differential equations. The stability of low-degree polynomials and Boolean circuits is established using tools from Gaussian and Boolean analysis—namely hypercontractivity and total influence, as well as a novel lower bound for random walks avoiding certain subsets, which we expect to be of independent interest. In the case of Boolean circuits, the result also makes use of Linial–Mansour–Nisan's classical theorem. Our techniques apply more broadly to low influence functions, and we expect that they may apply more generally.

computer science, theory & methods,mathematics, applied
Random optimization problems at fixed temperatures

Partha S. Dey,Grigory Terlov

2024-02-13

Abstract:This article considers a class of disordered mean-field combinatorial optimization problems. We focus on the Gibbs measure, where the inverse temperature does not vary with the size of the graph and the edge weights are sampled from a general distribution under mild assumptions. Our results consist of the Law of Large Numbers and Central Limit Theorems for the log-partition function, the weight of a typical configuration, and the Gibbs average in both quenched and annealed forms. We also derive quenched Poisson convergence for the size of the intersection of two independent samples, yielding replica symmetry of the model. Applications cover popular models from the literature, such as the Minimal Matching Problem, Traveling Salesman Problem, and Minimal Spanning Tree Problem, on a sequence of deterministic and random dense block graphs of increasing size.

Probability,Combinatorics
Iteratively sampling scheme for stochastic optimization with variable number sample path

Shuang Hao,Dali Zhang,Ming Dong

DOI: https://doi.org/10.1016/j.orl.2022.03.006

IF: 1.151

2022-01-01

Operations Research Letters

Abstract:Optimal search methods are proposed for solving optimization problems with analytically unobtainable objectives. This paper proposes a method by incorporating sampling schemes into the directional direct search with variable number sample path and investigates its effectiveness in solving stochastic optimization problems. We also explore the conditions on sample sizes at each iteration under which the convergence in probability can be guaranteed. Finally, a set of benchmark problems are numerically tested to show the effectiveness in different sampling schemes. (C) 2022 Elsevier B.V. All rights reserved.
Solution existence for a class of nonsmooth robust optimization problems

Nguyen Canh Hung,Thai Doan Chuong,Nguyen Le Hoang Anh

DOI: https://doi.org/10.1007/s10898-024-01450-9

2024-11-27

Journal of Global Optimization

Abstract:The main purpose of this paper is to investigate the existence of global optimal solutions for nonsmooth and nonconvex robust optimization problems. To do this, we first introduce a concept called extended tangency variety and show how a robust optimization problem can be transformed into a minimizing problem of the corresponding tangency variety. We utilize this concept together with a constraint qualification condition and the boundedness of the objective function to provide relationships among the concepts of robust properness, robust M-tamesness and robust Palais-Smale condition related to the considered problem. The obtained results are also employed to derive necessary and sufficient conditions for the existence of global optimal solutions to the underlying robust optimization problem.

mathematics, applied,operations research & management science
Statistical Methods for Optimization

L. Jeff Hong

DOI: https://doi.org/10.1002/9780470400531.eorms0828

2011-01-01

Abstract:AbstractMany statistical methods have been proposed to solve stochastic optimization problems where the objective functions are expected values of some random performances. In this article, we consider only ranking and selection: a collection of statistical procedures that intend to select the best solution from a finite (and often small) set of solutions, where the solutions can only be evaluated by running random computer experiments. We review two basic formulations of the problem and summarize how statistically valid procedures are designed to take into consideration the unique features of computer experiments.
The Value of Randomized Solutions in Mixed-Integer Distributionally Robust Optimization Problems

Erick Delage,Ahmed Saif

DOI: https://doi.org/10.1287/ijoc.2020.1042

IF: 3.288

2022-01-01

INFORMS Journal on Computing

Abstract:Randomized decision making refers to the process of making decisions randomly according to the outcome of an independent randomization device, such as a dice roll or a coin flip. The concept is unconventional, and somehow counterintuitive, in the domain of mathematical programming, in which deterministic decisions are usually sought even when the problem parameters are uncertain. However, it has recently been shown that using a randomized, rather than a deterministic, strategy in nonconvex distributionally robust optimization (DRO) problems can lead to improvements in their objective values. It is still unknown, though, what is the magnitude of improvement that can be attained through randomization or how to numerically find the optimal randomized strategy. In this paper, we study the value of randomization in mixed-integer DRO problems and show that it is bounded by the improvement achievable through its continuous relaxation. Furthermore, we identify conditions under which the bound is tight. We then develop algorithmic procedures, based on column generation, for solving both single- and two-stage linear DRO problems with randomization that can be used with both moment-based and Wasserstein ambiguity sets. Finally, we apply the proposed algorithm to solve three classical discrete DRO problems: the assignment problem, the uncapacitated facility location problem, and the capacitated facility location problem and report numerical results that show the quality of our bounds, the computational efficiency of the proposed solution method, and the magnitude of performance improvement achieved by randomized decisions. Summary of Contribution: In this paper, we present both theoretical results and algorithmic tools to identify optimal randomized strategies for discrete distributionally robust optimization (DRO) problems and evaluate the performance improvements that can be achieved when using them rather than classical deterministic strategies. On the theory side, we provide improvement bounds based on continuous relaxation and identify the conditions under which these bound are tight. On the algorithmic side, we propose a finitely convergent, two-layer, column-generation algorithm that iterates between identifying feasible solutions and finding extreme realizations of the uncertain parameter. The proposed algorithm was implemented to solve distributionally robust stochastic versions of three classical optimization problems and extensive numerical results are reported. The paper extends a previous, purely theoretical work of the first author on the idea of randomized strategies in nonconvex DRO problems by providing useful bounds and algorithms to solve this kind of problems.

computer science, interdisciplinary applications,operations research & management science
A Stochastic Approximation Framework for a Class of Randomized Optimization Algorithms

Jiaqiao Hu,Ping Hu,Hyeong Soo Chang

DOI: https://doi.org/10.1109/tac.2011.2158128

IF: 6.549

2012-01-01

IEEE Transactions on Automatic Control

Abstract:We study a class of random sampling-based algorithms for solving general non-differentiable optimization problems. These are iterative approaches that are based on sampling from and updating an underlying distribution function over the set of feasible solutions. In particular, we propose a novel and systematic framework to investigate the convergence and asymptotic convergence rates of these algorithms by exploiting their connections to the well-known stochastic approximation (SA) method. Such an SA framework unifies our understanding of these randomized algorithms and provides new insight into their design and implementation issues. Our preliminary numerical experiments indicate that new implementations of these algorithms based on the proposed framework may lead to improved performance over existing procedures.

automation & control systems,engineering, electrical & electronic
A Framework For Locally Convergent Random-Search Algorithms For Discrete Optimization Via Simulation

L. Jeff Hong,Barry L. Nelson

DOI: https://doi.org/10.1145/1276927.1276932

IF: 0.851

2007-01-01

ACM Transactions on Modeling and Computer Simulation

Abstract:The goal of this article is to provide a general framework for locally convergent random-search algorithms for stochastic optimization problems when the objective function is embedded in a stochastic simulation and the decision variables are integer ordered. The framework guarantees desirable asymptotic properties, including almost-sure convergence and known rate of convergence, for any algorithms that conform to its mild conditions. Within this framework, algorithm designers can incorporate sophisticated search schemes and complicated statistical procedures to design new algorithms.
Determining the Optimal Sampling Set Size for Random Search

Chenbo Zhu,Jie Xu,Chun-Hung Chen,Loo Hay Lee,Jianqiang Hu

DOI: https://doi.org/10.1109/wsc.2013.6721491

2013-01-01

Abstract:Random search is a core component of many well known simulation optimization algorithms such as nested partition and COMPASS. Given a fixed computation budget, a critical decision is how many solutions to sample from a search area, which directly determines the number of simulation replications for each solution assuming that each solution receives the same number of simulation replications. This is another instance of the exploration vs. exploitation tradeoff in simulation optimization. Modeling the performance profile of all solutions in the search area as a normal distribution, we propose a method to ( approximately) optimally determine the size of the sampling set and the number of simulation replications and use numerical experiments to demonstrate its performance.
Global optimization test problems based on random field composition

Ramses Sala,Niccolò Baldanzini,Marco Pierini

DOI: https://doi.org/10.48550/arXiv.1807.05096

2018-07-13

Optimization and Control

Abstract:The development and identification of effective optimization algorithms for non-convex real-world problems is a challenge in global optimization. Because theoretical performance analysis is difficult, and problems based on models of real-world systems are often computationally expensive, several artificial performance test problems and test function generators have been proposed for empirical comparative assessment and analysis of metaheuristic optimization algorithms. These test problems however often lack the complex function structures and forthcoming difficulties that can appear in real-world problems. This communication presents a method to systematically build test problems with various types and degrees of difficulty. By weighted composition of parameterized random fields, challenging test functions with tunable function features such as, variance contribution distribution, interaction order, and nonlinearity can be constructed. The method is described, and its applicability to optimization performance analysis is described by means of a few basic examples. The method aims to set a step forward in the systematic generation of global optimization test problems, which could lead to a better understanding of the performance of optimization algorithms on problem types with particular characteristics. On request an introductive MATLAB implementation of a test function generator based on the presented method is available.
Solving the Satisfiability Problem by Using Randomized Approach

LC WU,CY TANG

DOI: https://doi.org/10.1016/0020-0190(92)90177-w

IF: 0.851

1992-01-01

Information Processing Letters

Abstract:In this paper we propose a randomized algorithm which can solve the satisfiability problem with the probability of failure not exceeding ∈ in polynomial average time.

The existence of random solutions to random optimization problems

A novel stochastic method with modified extremal optimization and nearest neighbor search for hard combinatorial problems

On the stability of solutions to random optimization problems under small perturbations

Studies on Some Modern Optimization Algorithms

A Bound on the Expected Optimality of Random Feasible Solutions to Combinatorial Optimization Problems

Algorithm-independent bounds on complex optimization through the statistics of marginal optima

Some Remarks on the Optimal Level of Randomization in Global Optimization

Simulation-Based Discrete Optimization of Stochastic Discrete Event Systems Subject to Non Closed-Form Constraints.

Existence theorems for optimal solutions in semi-algebraic optimization

A Note On The Popularity of Stochastic Optimization Algorithms in Different Fields: A Quantitative Analysis from 2007 to 2017

Hardness of Random Optimization Problems for Boolean Circuits, Low-Degree Polynomials, and Langevin Dynamics

Random optimization problems at fixed temperatures

Iteratively sampling scheme for stochastic optimization with variable number sample path

Solution existence for a class of nonsmooth robust optimization problems

Statistical Methods for Optimization

The Value of Randomized Solutions in Mixed-Integer Distributionally Robust Optimization Problems

A Stochastic Approximation Framework for a Class of Randomized Optimization Algorithms

A Framework For Locally Convergent Random-Search Algorithms For Discrete Optimization Via Simulation

Determining the Optimal Sampling Set Size for Random Search

Global optimization test problems based on random field composition

Solving the Satisfiability Problem by Using Randomized Approach