Abstract:Sum of squares (SOS) optimization is a powerful technique for solving problems where the positivity of a polynomials must be enforced. The common approach to solve an SOS problem is by relaxation to a Semidefinite Program (SDP). The main advantage of this transormation is that SDP is a convex problem for which efficient solvers are readily available. However, while considerable progress has been made in recent years, the standard approaches for solving SDPs are still known to scale poorly. Our goal is to devise an approach that can handle larger, more complex problems than is currently possible. The challenge indeed lies in how SDPs are commonly solved. State-Of-The-Art approaches rely on the interior point method, which requires the factorization of large matrices. We instead propose an approach inspired by polynomial neural networks, which exhibit excellent performance when optimized using techniques from the deep learning toolbox. In a somewhat counter-intuitive manner, we replace the convex SDP formulation with a non-convex, unconstrained, and \emph{over parameterized} formulation, and solve it using a first order optimization method. It turns out that this approach can handle very large problems, with polynomials having over four million coefficients, well beyond the range of current SDP-based approaches. Furthermore, we highlight theoretical and practical results supporting the experimental success of our approach in avoiding spurious local minima, which makes it amenable to simple and fast solutions based on gradient descent. In all the experiments, our approach had always converged to a correct global minimum, on general (non-sparse) polynomials, with running time only slightly higher than linear in the number of polynomial coefficients, compared to higher than quadratic in the number of coefficients for SDP-based methods.

Shape-Constrained Regression using Sum of Squares Polynomials

Shape-constrained Estimation in Functional Regression with Bernstein Polynomials

A practical, fast method for solving sum-of-squares problems for very large polynomials

S-SOS: Stochastic Sum-Of-Squares for Parametric Polynomial Optimization

Incorporating Shape Knowledge into Regression Models

Convex Regression in Multidimensions: Suboptimality of Least Squares Estimators

Sparse Sub-gaussian Random Projections for Semidefinite Programming Relaxations

A Fast Algorithm for Multidimensional Ellipsoid-Specific Fitting by Minimizing a New Defined Vector Norm of Residuals Using Semidefinite Programming

Sum-of-squares hierarchies for polynomial optimization and the Christoffel-Darboux kernel

Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations

Robust Learning from Noisy Side-information by Semidefinite Programming

New SDP Roundings and Certifiable Approximation for Cubic Optimization

Linear-memory and Decomposition-invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes

Nonparametric Shape-restricted Regression

Trust--Region Problems with Linear Inequality Constraints: Exact SDP Relaxation, Global Optimality and Robust Optimization

A conditional gradient homotopy method with applications to Semidefinite Programming

Improving Efficiency and Scalability of Sum of Squares Optimization: Recent Advances and Limitations

Suboptimality bounds for trace-bounded SDPs enable a faster and scalable low-rank SDP solver SDPLR+

An Algebraic Approach to a Class of Rank-Constrained Semi-Definite Programs With Applications

Shape Constraints in Symbolic Regression using Penalized Least Squares

The Geometry and Well-Posedness of Sparse Regularized Linear Regression