Abstract:Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an 'all-in-one' Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an 'as-needed' basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.

Global Convergence Analysis of Decomposition Methods for Support Vector Regression

Convergence of Decomposition Methods for Support Vector Machines

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Two New Decomposition Algorithms for Training Bound-Constrained Support Vector Machines

A decomposition algorithm for solving large-scale quadratic programming problems

A Feasible Descent Decomposition Algorithm for Large-scale Quadratic Programming

A New Decomposition Method for Fully Non-Separable Large-Scale Problem

A DECOMPOSITION AUGMENTED LAGRANGIAN METHOD FOR LOW-RANK SEMIDEFINITE PROGRAMMING

A Surrogate-Assisted Variable Grouping Algorithm for General Large Scale Global Optimization Problems

Convergence Analysis of General Decomposition Algorithm for Solving General Constrained Variational Inequality Problems

Distributed Sequential Quadratic Programming with Overlapping Graph Decomposition and Exact Augmented Lagrangian

A DECOMPOSITION METHOD FOR CONVEX MINIMIZATION PROBLEMS AND ITS APPLICATION

Asymptotic Results of Stochastic Decomposition for Two-Stage Stochastic Quadratic Programming

A First-Order Decomposition Algorithm for Training Bound-Constrained Support Vector Machines.

An Eigenvalue Decomposition Method for Low-rank and Non-convex Quadratically Constrained Quadratic Programming

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines

A Decomposition Algorithm for Two-Stage Stochastic Programs with Nonconvex Recourse Functions

A Globally Convergent Stabilized SQP Method

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines

A Partially Parallel Splitting Method for Multiple-Block Separable Convex Programming with Applications to Robust Pca

CONVERGENCE ANALYSIS OF DISCRETE-TIME NEURAL NETWORK FOR SOLVING QUADRATIC PROGRAMMING PROBLEMS