Abstract:Quadratic programming is a ubiquitous prototype in convex programming. Many combinatorial optimizations on graphs and machine learning problems can be formulated as quadratic programming; for example, Support Vector Machines (SVMs). Linear and kernel SVMs have been among the most popular models in machine learning over the past three decades, prior to the deep learning era. Generally, a quadratic program has an input size of $\Theta(n^2)$, where $n$ is the number of variables. Assuming the Strong Exponential Time Hypothesis ($\textsf{SETH}$), it is known that no $O(n^{2-o(1)})$ algorithm exists (Backurs, Indyk, and Schmidt, NIPS'17). However, problems such as SVMs usually feature much smaller input sizes: one is given $n$ data points, each of dimension $d$, with $d \ll n$. Furthermore, SVMs are variants with only $O(1)$ linear constraints. This suggests that faster algorithms are feasible, provided the program exhibits certain underlying structures. In this work, we design the first nearly-linear time algorithm for solving quadratic programs whenever the quadratic objective has small treewidth or admits a low-rank factorization, and the number of linear constraints is small. Consequently, we obtain a variety of results for SVMs: * For linear SVM, where the quadratic constraint matrix has treewidth $\tau$, we can solve the corresponding program in time $\widetilde O(n\tau^{(\omega+1)/2}\log(1/\epsilon))$; * For linear SVM, where the quadratic constraint matrix admits a low-rank factorization of rank-$k$, we can solve the corresponding program in time $\widetilde O(nk^{(\omega+1)/2}\log(1/\epsilon))$; * For Gaussian kernel SVM, where the data dimension $d = \Theta(\log n)$ and the squared dataset radius is small, we can solve it in time $O(n^{1+o(1)}\log(1/\epsilon))$. We also prove that when the squared dataset radius is large, then $\Omega(n^{2-o(1)})$ time is required.

On Computing the Minima of Quadratic Forms (Preliminary Report)

An approximation theory of matrix rank minimization and its application to quadratic equations

On Sketching Quadratic Forms

Complexity of Minimizing Regularized Convex Quadratic Functions

Toward the Optimal Construction of a Loss Function Without Spurious Local Minima for Solving Quadratic Equations

Computable Representation of the Cone of Nonnegative Quadratic Forms over A General Second-Order Cone and Its Application to Completely Positive Programming

Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems

On the complexity of finding a local minimizer of a quadratic function over a polytope

A Fast Algorithm for Multidimensional Ellipsoid-Specific Fitting by Minimizing a New Defined Vector Norm of Residuals Using Semidefinite Programming

Computing The Minimum Distance Between Two Ellipsoids

On a minimization problem of the maximum generalized eigenvalue: properties and algorithms

Solving Almost All Systems of Random Quadratic Equations.

Faster Algorithms for Structured Linear and Kernel Support Vector Machines

Quadratization of ODEs: Monomial vs. Non-Monomial

Minimization of Quadratic Binary Functional with Additive Connection Matrix

Spurious local minima in nonconvex sum-of-squares optimization

Positive values of non-homogeneous quadratic forms of type (1,4): A conjecture of Bambah, Dumir and Hans-Gill

A Family of Iteration Functions for General Linear Systems

On the Geometric Analysis of a Quartic–quadratic Optimization Problem under a Spherical Constraint

A General Analysis Framework of Lower Complexity Bounds for Finite-Sum Optimization

Least Squares Solution with the Minimum-Norm to General Matrix Equations Via Iteration.