Abstract:Quadratic programming is a ubiquitous prototype in convex programming. Many combinatorial optimizations on graphs and machine learning problems can be formulated as quadratic programming; for example, Support Vector Machines (SVMs). Linear and kernel SVMs have been among the most popular models in machine learning over the past three decades, prior to the deep learning era. Generally, a quadratic program has an input size of $\Theta(n^2)$, where $n$ is the number of variables. Assuming the Strong Exponential Time Hypothesis ($\textsf{SETH}$), it is known that no $O(n^{2-o(1)})$ algorithm exists (Backurs, Indyk, and Schmidt, NIPS'17). However, problems such as SVMs usually feature much smaller input sizes: one is given $n$ data points, each of dimension $d$, with $d \ll n$. Furthermore, SVMs are variants with only $O(1)$ linear constraints. This suggests that faster algorithms are feasible, provided the program exhibits certain underlying structures. In this work, we design the first nearly-linear time algorithm for solving quadratic programs whenever the quadratic objective has small treewidth or admits a low-rank factorization, and the number of linear constraints is small. Consequently, we obtain a variety of results for SVMs: * For linear SVM, where the quadratic constraint matrix has treewidth $\tau$, we can solve the corresponding program in time $\widetilde O(n\tau^{(\omega+1)/2}\log(1/\epsilon))$; * For linear SVM, where the quadratic constraint matrix admits a low-rank factorization of rank-$k$, we can solve the corresponding program in time $\widetilde O(nk^{(\omega+1)/2}\log(1/\epsilon))$; * For Gaussian kernel SVM, where the data dimension $d = \Theta(\log n)$ and the squared dataset radius is small, we can solve it in time $O(n^{1+o(1)}\log(1/\epsilon))$. We also prove that when the squared dataset radius is large, then $\Omega(n^{2-o(1)})$ time is required.

Sparse Quadratic Logistic Regression in Sub-quadratic Time

Recovery Guarantees for Quadratic Tensors with Sparse Observations

Subspace Quadratic Regularization Method for Group Sparse Multinomial Logistic Regression

Sparse Least Squares Support Vector Machine for Function Estimation

Efficient and robust high-dimensional sparse logistic regression via nonlinear primal-dual hybrid gradient algorithms

Faster Algorithms for Structured Linear and Kernel Support Vector Machines

Online Kernel Learning with a Near Optimal Sparsity Bound

Robust Sparse Regression with Non-Isotropic Designs

Feature Adaptation for Sparse Linear Regression

High-dimensional classification by sparse logistic regression

Clinical Risk Prediction With Multilinear Sparse Logistic Regression

Globally Convergent Accelerated Algorithms for Multilinear Sparse Logistic Regression with $\ell_0$-constraints

Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space

Beyond L2-Loss Functions for Learning Sparse Models

Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach

Quadratic Approximation Greedy Pursuit for Cardinality-Constrained Sparse Learning

Sparse recovery from quadratic equations, part II: hardness and incoherence

An improved IRLS algorithm for sparse recovery with intra-block correlation

Computational-Statistical Gaps for Improper Learning in Sparse Linear Regression

L0 Regularized Logistic Regression for Large-Scale Data

Optimal Robust Linear Regression in Nearly Linear Time