Abstract:Stochastic First-Order (SFO) methods have been a cornerstone in addressing a broad spectrum of modern machine learning (ML) challenges. However, their efficacy is increasingly questioned, especially in large-scale applications where empirical evidence indicates potential performance limitations. In response, this paper proposes an innovative method specifically designed for kernel support vector machines (SVMs). This method not only achieves faster convergence per iteration but also exhibits enhanced scalability when compared to conventional SFO techniques. Diverging from traditional sample average approximation strategies that typically frame kernel SVM as an 'all-in-one' Quadratic Program (QP), our approach adopts adaptive sampling. This strategy incrementally refines approximation accuracy on an 'as-needed' basis. Crucially, this approach also inspires a decomposition-based algorithm, effectively decomposing parameter selection from error estimation, with the latter being independently determined for each data point. To exploit the quadratic nature of the kernel matrix, we introduce a stochastic conjugate subgradient method. This method preserves many benefits of first-order approaches while adeptly handling both nonlinearity and non-smooth aspects of the SVM problem. Thus, it extends beyond the capabilities of standard SFO algorithms for non-smooth convex optimization. The convergence rate of this novel method is thoroughly analyzed within this paper. Our experimental results demonstrate that the proposed algorithm not only maintains but potentially exceeds the scalability of SFO methods. Moreover, it significantly enhances both speed and accuracy of the optimization process.

Conformal Transformation of Kernel Functions: A Data-Dependent Way to Improve Support Vector Machine Classifiers

Conformal Transformation of Kernel Functions

Conformal Transformation of Kernels: A Geometric Perspective on Text Classification

Improving Support Vector Machine Classifiers By Modifying Kernel Functions

An information-geometrical method for improving the performance of support vector machine classifiers

Support Vector Machines Based on Information Geometry

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Distance-Based Classifier Via the Kernel Trick.

Scaling the kernel function to improve performance of the support vector machine

A support vector data description algorithm based on an adaptive kernel function

A geometrical method to improve performance of the support vector machine.

Optimized Kernel-Based Conformal Predictor for Online Fault Detection

Optimized conformal classification using gradient descent approximation

Conformalized Kernel Ridge Regression

A new trigonometric kernel function for support vector machine

Optimizing Kernel Transformations to Handle Binary Class Imbalanced Dataset Classification

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines

Conformalization of Sparse Generalized Linear Models

Improving classification performance by feature space transformations and model selection

Conformal Prediction using Conditional Histograms

A New Class of Bessel Kernel Functions for Support Vector Machine