Abstract:Least squares support vector machines are a commonly used supervised learning method for nonlinear regression and classification. They can be implemented in either their primal or dual form. The latter requires solving a linear system, which can be advantageous as an explicit mapping of the data to a possibly infinite-dimensional feature space is avoided. However, for large-scale applications, current low-rank approximation methods can perform inadequately. For example, current methods are probabilistic due to their sampling procedures, and/or suffer from a poor trade-off between the ranks and approximation power. In this paper, a recursive Bayesian filtering framework based on tensor networks and the Kalman filter is presented to alleviate the demanding memory and computational complexities associated with solving large-scale dual problems. The proposed method is iterative, does not require explicit storage of the kernel matrix, and allows the formulation of early stopping conditions. Additionally, the framework yields confidence estimates of obtained models, unlike alternative methods. The performance is tested on two regression and three classification experiments, and compared to the Nyström and fixed size LS-SVM methods. Results show that our method can achieve high performance and is particularly useful when alternative methods are computationally infeasible due to a slowly decaying kernel matrix spectrum.

Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates

Using Svm To Model And Control Nonlinear Dynamical Systems

Sparse Least Squares Support Vector Machine for Function Estimation

Stochastic Sub-Sampled Newton Method with Variance Reduction

A structured diagonal Hessian approximation method with evaluation complexity analysis for nonlinear least squares

Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization

A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data

Tensor Network Kalman Filtering for Large-Scale LS-SVMs

A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Estimating linear covariance models with numerical nonlinear algebra

Fast Sparse Least-Squares Regression with Non-Asymptotic Guarantees

A Hessian for Gaussian Mixture Likelihoods in Nonlinear Least Squares

Incremental Gauss--Newton Methods with Superlinear Convergence Rates

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

Sequential alternating least squares for solving high dimensional linear Hamilton-Jacobi-Bellman equation

Optimizing Variational Physics-Informed Neural Networks Using Least Squares

Levenberg–Marquardt Method Based on Probabilistic Jacobian Models for Nonlinear Equations

Adaptive Stochastic Gradient Descent on the Grassmannian for Robust Low-Rank Subspace Recovery

Stochastic gradient descent for linear least squares problems with partially observed data