Abstract:Most of machine learning approaches have stemmed from the application of minimizing the mean squared distance principle, based on the computationally efficient quadratic optimization methods. However, when faced with high-dimensional and noisy data, the quadratic error functionals demonstrated many weaknesses including high sensitivity to contaminating factors and dimensionality curse. Therefore, a lot of recent applications in machine learning exploited properties of non-quadratic error functionals based on L1 norm or even sub-linear potentials corresponding to quasinorms Lp (0<p<1). The back side of these approaches is increase in computational cost for optimization. Till so far, no approaches have been suggested to deal with arbitrary error functionals, in a flexible and computationally efficient framework. In this paper, we develop a theory and basic universal data approximation algorithms (k-means, principal components, principal manifolds and graphs, regularized and sparse regression), based on piece-wise quadratic error potentials of subquadratic growth (PQSQ potentials). We develop a new and universal framework to minimize arbitrary sub-quadratic error potentials using an algorithm with guaranteed fast convergence to the local or global error minimum. The theory of PQSQ potentials is based on the notion of the cone of minorant functions, and represents a natural approximation formalism based on the application of min-plus algebra. The approach can be applied in most of existing machine learning methods, including methods of data approximation and regularized and sparse regression, leading to the improvement in the computational cost/accuracy trade-off. We demonstrate that on synthetic and real-life datasets PQSQ-based machine learning methods achieve orders of magnitude faster computational performance than the corresponding state-of-the-art methods, having similar or better approximation accuracy.

Learning Theory Viewpoint of Approximation by Positive Linear Operators

Analysis of Approximation by Linear Operators on VariableLρp(·)Spaces and Applications in Learning Theory

Approximation Analysis of Learning Algorithms for Support Vector Regression and Quantile Regression

On the Approximation of Kernel functions

Learning Theory Estimates for Coefficient-Based Regularized Regression

Full length article: Approximation by multivariate Bernstein-Durrmeyer operators and learning rates of least-squares regularized regression with multivariate polynomial kernels

Approximation beats concentration? An approximation view on inference with smooth radial kernels

Learning Lipschitz Operators with respect to Gaussian Measures with Near-Optimal Sample Complexity

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Learning Errors of Linear Programming Support Vector Regression

Least Squares Approximations in Linear Statistical Inverse Learning Problems

Approximation in Hilbert spaces of the Gaussian and related analytic kernels

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Sparse Approximation to a Key Algorithm of Learning Theory

Approximation of discontinuous functions by positive linear operators. A probabilistic approach

Robust Kernel Approximation For Classification

Piece-wise quadratic approximations of arbitrary error functions for fast and robust machine learning

Approximate Kernel Partial Least Squares

Kernel Based Approximation in Sobolev Spaces with Radial Basis Functions

On the Probabilistic Approximation in Reproducing Kernel Hilbert Spaces

Koopman Theory and Linear Approximation Spaces