A survey of hidden convex optimization

Yong Xia

DOI: https://doi.org/10.48550/arXiv.1902.10921

2019-02-28

Abstract:Motivated by the fact that not all nonconvex optimization problems are difficult to solve, we survey in this paper three widely-used ways to reveal the hidden convex structure for different classes of nonconvex optimization problems. Finally, ten open problems are raised.

Optimization and Control

What problem does this paper attempt to address?

Convexification and Monotone Optimization

Xiaoling Sun,Jianling Li,Duan Li

DOI: https://doi.org/10.1007/0-387-26771-9_9

2005-01-01

Abstract:Monotone maximization is a global optimization problem that maximizes an increasing function subject to increasing constraints. Due to the often existence of multiple local optimal solutions, finding a global optimal solution of such a problem is computationally difficult. In this survey paper, we summarize global solution methods for the monotone optimization problem. In particular, we propose a unified framework for the recent progress on convexification methods for the monotone optimization problem. Suggestions for further research are also presented.
Stochastic Optimization under Hidden Convexity

Ilyas Fatkhullin,Niao He,Yifan Hu

2023-12-30

Abstract:In this work, we consider constrained stochastic optimization problems under hidden convexity, i.e., those that admit a convex reformulation via non-linear (but invertible) map $c(\cdot)$. A number of non-convex problems ranging from optimal control, revenue and inventory management, to convex reinforcement learning all admit such a hidden convex structure. Unfortunately, in the majority of applications considered, the map $c(\cdot)$ is unavailable or implicit; therefore, directly solving the convex reformulation is not possible. On the other hand, the stochastic gradients with respect to the original variable are often easy to obtain. Motivated by these observations, we examine the basic projected stochastic (sub-) gradient methods for solving such problems under hidden convexity. We provide the first sample complexity guarantees for global convergence in smooth and non-smooth settings. Additionally, in the smooth setting, we improve our results to the last iterate convergence in terms of function value gap using the momentum variant of projected stochastic gradient descent.

Optimization and Control,Computational Complexity
Non-convex Optimization for Machine Learning

Prateek Jain,Purushottam Kar

DOI: https://doi.org/10.1561/2200000058

2017-01-01

Foundations and Trends® in Machine Learning

Abstract:A vast majority of machine learning algorithms train their models and perform inference by solving optimization problems. In order to capture the learning and prediction problems accurately, structural constraints such as sparsity or low rank are frequently imposed or else the objective itself is designed to be a non-convex function. This is especially true of algorithms that operate in high-dimensional spaces or that train non-linear models such as tensor models and deep networks. The freedom to express the learning problem as a non-convex optimization problem gives immense modeling power to the algorithm designer, but often such problems are NP-hard to solve. A popular workaround to this has been to relax non-convex problems to convex ones and use traditional methods to solve the (convex) relaxed optimization problems. However this approach may be lossy and nevertheless presents significant challenges for large scale optimization. On the other hand, direct approaches to non-convex optimization have met with resounding success in several domains and remain the methods of choice for the practitioner, as they frequently outperform relaxation-based techniques – popular heuristics include projected gradient descent and alternating minimization. However, these are often poorly understood in terms of their convergence and other properties. This monograph presents a selection of recent advances that bridge a long-standing gap in our understanding of these heuristics. We hope that an insight into the inner workings of these methods will allow the reader to appreciate the unique marriage of task structure and generative models that allow these heuristic techniques to (provably) succeed. The monograph will lead the reader through several widely used nonconvex optimization techniques, as well as applications thereof. The goal of this monograph is to both, introduce the rich literature in this area, as well as equip the reader with the tools and techniques needed to analyze these simple procedures for non-convex problems.
Local Optimality Conditions for a Class of Hidden Convex Optimization

Mengmeng Song,Yong Xia,Hongying Liu

DOI: https://doi.org/10.48550/arXiv.2109.03110

2021-09-07

Optimization and Control

Abstract:Hidden convex optimization is such a class of nonconvex optimization problems that can be globally solved in polynomial time via equivalent convex programming reformulations. In this paper, we focus on checking local optimality in hidden convex optimization. We first introduce a class of hidden convex optimization problems by jointing the classical nonconvex trust-region subproblem (TRS) with convex optimization (CO), and then present a comprehensive study on local optimality conditions. In order to guarantee the existence of a necessary and sufficient condition for local optimality, we need more restrictive assumptions. To our surprise, while (TRS) has at most one local non-global minimizer and (CO) has no local non-global minimizer, their joint problem could have more than one local non-global minimizer.
A Survey on Probabilistically Constrained Optimization Problems

Xiaodi Bai,Xiaojin Zheng,Xiaoling Sun

DOI: https://doi.org/10.3934/naco.2012.2.767

2012-01-01

Numerical Algebra, Control and Optimization

Abstract:Probabilistically constrained optimization problems are an important class of stochastic programming problems with wide applications in finance, management and engineering planning. In this paper, we summarize some important solution methods including convex approximation, DC approach, scenario approach and integer programming approach. We also discuss some future research perspectives on the probabilistically constrained optimization problems.
Convex Discrete Optimization

Shmuel Onn

DOI: https://doi.org/10.48550/arXiv.math/0703575

2007-03-20

Abstract:We develop an algorithmic theory of convex optimization over discrete sets. Using a combination of algebraic and geometric tools we are able to provide polynomial time algorithms for solving broad classes of convex combinatorial optimization problems and convex integer programming problems in variable dimension. We discuss some of the many applications of this theory including to quadratic programming, matroids, bin packing and cutting-stock problems, vector partitioning and clustering, multiway transportation problems, and privacy and confidential statistical data disclosure. Highlights of our work include a strongly polynomial time algorithm for convex and linear combinatorial optimization over any family presented by a membership oracle when the underlying polytope has few edge-directions; a new theory of so-termed n-fold integer programming, yielding polynomial time solution of important and natural classes of convex and linear integer programming problems in variable dimension; and a complete complexity classification of high dimensional transportation problems, with practical applications to fundamental problems in privacy and confidential statistical data disclosure.

Optimization and Control,Computational Complexity,Discrete Mathematics,Combinatorics
Solving a Class of Convex Stochastic Programs Via Monte Carlo Simulation

Maojun Zhang,Zunquan Xia,Mingzheng Wang,Jiangxia Nan

DOI: https://doi.org/10.3969/j.issn.1007-6093.2009.02.004

2009-01-01

Abstract:Algorithms based on Monte Carlo sampling both of objective and constraint functions are presented for solving convex stochastic programs, and their convergence theorems are given. They are employed to deal with portfolio optimization problems with a semi-variance constraint. Numerical experiments are also given.
Convexification, Concavification and Monotonization in Global Optimization

D. Li,X.L. Sun,M.P. Biswal,F. Gao

DOI: https://doi.org/10.1023/A:1013313901854

IF: 4.82

2001-01-01

Annals of Operations Research

Abstract:We show in this paper that via certain convexification, concavification and monotonization schemes a nonconvex optimization problem over a simplex can be always converted into an equivalent better-structured nonconvex optimization problem, e.g., a concave optimization problem or a D.C. programming problem, thus facilitating the search of a global optimum by using the existing methods in concave minimization and D.C. programming. We first prove that a monotone optimization problem (with a monotone objective function and monotone constraints) can be transformed into a concave minimization problem over a convex set or a D.C. programming problem via p th power transformation. We then prove that a class of nonconvex minimization problems can be always reduced to a monotone optimization problem, thus a concave minimization problem or a D.C. programming problem.
Disciplined Multi-Convex Programming

Xinyue Shen,Steven Diamond,Madeleine Udell,Yuantao Gu,Stephen Boyd

DOI: https://doi.org/10.1109/ccdc.2017.7978647

2017-01-01

Abstract:A multi-convex optimization problem is one in which the variables can be partitioned into sets over which the problem is convex when the other variables are fixed. Multi-convex problems are generally solved approximately using variations on alternating or cyclic minimization. Multi-convex problems arise in many applications, such as non-negative matrix factorization, generalized low rank models, and structured control synthesis, to name just a few. In most applications to date the multi-convexity is simple to verify by hand. In this paper we study the automatic detection and verification of multi-convexity using the ideas of disciplined convex programming. We describe an implementation of our proposed method that detects and verifies multi-convexity, and then invokes one of the general solution methods.
Optimization Methods for Inverse Problems

Nan Ye,Farbod Roosta-Khorasani,Tiangang Cui

DOI: https://doi.org/10.48550/arXiv.1712.00154

2017-12-01

Abstract:Optimization plays an important role in solving many inverse problems. Indeed, the task of inversion often either involves or is fully cast as a solution of an optimization problem. In this light, the mere non-linear, non-convex, and large-scale nature of many of these inversions gives rise to some very challenging optimization problems. The inverse problem community has long been developing various techniques for solving such optimization tasks. However, other, seemingly disjoint communities, such as that of machine learning, have developed, almost in parallel, interesting alternative methods which might have stayed under the radar of the inverse problem community. In this survey, we aim to change that. In doing so, we first discuss current state-of-the-art optimization methods widely used in inverse problems. We then survey recent related advances in addressing similar challenges in problems faced by the machine learning community, and discuss their potential advantages for solving inverse problems. By highlighting the similarities among the optimization challenges faced by the inverse problem and the machine learning communities, we hope that this survey can serve as a bridge in bringing together these two communities and encourage cross fertilization of ideas.

Optimization and Control
Hidden convexity, optimization, and algorithms on rotation matrices

Akshay Ramachandran,Kevin Shu,Alex L. Wang

2024-04-30

Abstract:This paper studies hidden convexity properties associated with constrained optimization problems over the set of rotation matrices $\text{SO}(n)$. Such problems are nonconvex due to the constraint $X \in \text{SO}(n)$. Nonetheless, we show that certain linear images of $\text{SO}(n)$ are convex, opening up the possibility for convex optimization algorithms with provable guarantees for these problems. Our main technical contributions show that any two-dimensional image of $\text{SO}(n)$ is convex and that the projection of $\text{SO}(n)$ onto its strict upper triangular entries is convex. These results allow us to construct exact convex reformulations for constrained optimization problems over $\text{SO}(n)$ with a single constraint or with constraints defined by low-rank matrices. Both of these results are optimal in a formal sense.

Optimization and Control
Non-convex mixed-integer nonlinear programming: A survey

Samuel Burer,Adam N. Letchford

DOI: https://doi.org/10.1016/j.sorms.2012.08.001

2012-07-01

Surveys in Operations Research and Management Science

Abstract:A wide range of problems arising in practical applications can be formulated as Mixed-Integer Nonlinear Programs (MINLPs). For the case in which the objective and constraint functions are convex, some quite effective exact and heuristic algorithms are available. When non-convexities are present, however, things become much more difficult, since then even the continuous relaxation is a global optimization problem. We survey the literature on non-convex MINLPs, discussing applications, algorithms, and software. Special attention is paid to the case in which the objective and constraint functions are quadratic.
Subspace Methods for Nonlinear Optimization

Xin Liu,Zaiwen Wen,Ya-Xiang Yuan

DOI: https://doi.org/10.4208/csiam-am.so-2021-0016

2021-01-01

CSIAM Transactions on Applied Mathematics

Abstract:Subspace techniques such as Krylov subspace methods have been well known and extensively used in numerical linear algebra. They are also ubiquitous and becoming indispensable tools in nonlinear optimization due to their ability to handle large scale problems. There are generally two types of principals: i) the decision variable is updated in a lower dimensional subspace; ii) the objective function or constraints are approximated in a certain smaller subspace of their domain. The key ingredients are the constructions of suitable subspaces and subproblems according to the specific structures of the variables and functions such that either the exact or inexact solutions of subproblems are readily available and the corresponding computational cost is significantly reduced. A few relevant techniques include but not limited to direct combinations, block coordinate descent, active sets, limited-memory, Anderson acceleration, subspace correction, sampling and sketching. This paper gives a comprehensive survey on the subspace methods and their recipes in unconstrained and constrained optimization, nonlinear least squares problem, sparse and low rank optimization, linear and nonlinear eigenvalue computation, semidefinite programming, stochastic optimization and etc. In order to provide helpful guidelines, we emphasize on high level concepts for the development and implementation of practical algorithms from the subspace framework.
A Convexification Method for a Class of Global Optimization Problems with Applications to Reliability Optimization

X. L. Sun,K.I.M. McKinnon,D. Li

DOI: https://doi.org/10.1023/a:1011962605464

2001-01-01

Journal of Global Optimization

Abstract:A convexification method is proposed for solving a class of global optimization problems with certain monotone properties. It is shown that this class of problems can be transformed into equivalent concave minimization problems using the proposed convexification schemes. An outer approximation method can then be used to find the global solution of the transformed problem. Applications to mixed-integer nonlinear programming problems arising in reliability optimization of complex systems are discussed and satisfactory numerical results are presented.
Generalized Polyhedral Convex Optimization Problems

Nguyen Ngoc Luan,Jen-Chih Yao

DOI: https://doi.org/10.48550/arXiv.1709.10227

2017-09-29

Abstract:Generalized polyhedral convex optimization problems in locally convex Hausdorff topological vector spaces are studied systematically in this paper. We establish solution existence theorems, necessary and sufficient optimality conditions, weak and strong duality theorems. In particular, we show that the dual problem has the same structure as the primal problem, and the strong duality relation holds under three different sets of conditions.

Optimization and Control
Convex Combinatorial Optimization

Shmuel Onn,Uriel G. Rothblum

DOI: https://doi.org/10.48550/arXiv.math/0309083

2003-09-05

Abstract:We introduce the convex combinatorial optimization problem, a far reaching generalization of the standard linear combinatorial optimization problem. We show that it is strongly polynomial time solvable over any edge-guaranteed family, and discuss several applications.

Combinatorics,Discrete Mathematics,Optimization and Control
A Survey of Optimization Methods From a Machine Learning Perspective

Shiliang Sun,Zehui Cao,Han Zhu,Jing Zhao

DOI: https://doi.org/10.1109/tcyb.2019.2950779

IF: 11.8

2020-08-01

IEEE Transactions on Cybernetics

Abstract:Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this article, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Finally, we explore and give some challenges and open problems for the optimization in machine learning.

automation & control systems,computer science, cybernetics, artificial intelligence
Introduction to Online Convex Optimization

Elad Hazan

2023-08-06

Abstract:This manuscript portrays optimization as a process. In many practical applications the environment is so complex that it is infeasible to lay out a comprehensive theoretical model and use classical algorithmic theory and mathematical optimization. It is necessary as well as beneficial to take a robust approach, by applying an optimization method that learns as one goes along, learning from experience as more aspects of the problem are observed. This view of optimization as a process has become prominent in varied fields and has led to some spectacular success in modeling and systems that are now part of our daily lives.

Machine Learning,Optimization and Control
Conditional Infimum and Hidden Convexity in Optimization

Jean-Philippe Chancelier,Michel de Lara

DOI: https://doi.org/10.48550/arXiv.2104.05266

2021-04-12

Abstract:Detecting hidden convexity is one of the tools to address nonconvex minimization problems. After giving a formal definition of hidden convexity, we introduce the notion of conditional infimum, as it will prove instrumental in detecting hidden convexity. We develop the theory of the conditional infimum, and we establish a tower property, relevant for minimization problems. Thus equipped, we provide a sufficient condition for hidden convexity in nonconvex minimization problems. We illustrate our result on nonconvex quadratic minimization problems. We conclude with perspectives for using the conditional infimum in relation to the so-called S-procedure, to couplings and conjugacies, and to lower bound convex programs.

Optimization and Control

A survey of hidden convex optimization

Convexification and Monotone Optimization

Stochastic Optimization under Hidden Convexity

Non-convex Optimization for Machine Learning

Local Optimality Conditions for a Class of Hidden Convex Optimization

A Survey on Probabilistically Constrained Optimization Problems

Convex Discrete Optimization

Solving a Class of Convex Stochastic Programs Via Monte Carlo Simulation

Convexification, Concavification and Monotonization in Global Optimization

Disciplined Multi-Convex Programming

Optimization Methods for Inverse Problems

Hidden convexity, optimization, and algorithms on rotation matrices

Non-convex mixed-integer nonlinear programming: A survey

Subspace Methods for Nonlinear Optimization

A Convexification Method for a Class of Global Optimization Problems with Applications to Reliability Optimization

Generalized Polyhedral Convex Optimization Problems

Convex Combinatorial Optimization

A Survey of Optimization Methods From a Machine Learning Perspective

Introduction to Online Convex Optimization

Conditional Infimum and Hidden Convexity in Optimization