Abstract:Kernel method has been developed as one of the standard approaches for nonlinear learning, which however, does not scale to large data set due to its quadratic complexity in the number of samples. A number of kernel approximation methods have thus been proposed in the recent years, among which the random features method gains much popularity due to its simplicity and direct reduction of nonlinear problem to a linear one. Different random feature functions have since been proposed to approximate a variety of kernel functions. Among them the Random Binning (RB) feature, proposed in the first random-feature paper [21], has drawn much less attention than the Random Fourier (RF) feature proposed also in [21]. In this work, we observe that the RB features, with right choice of optimization solver, could be orders-of-magnitude more efficient than other random features and kernel approximation methods under the same requirement of accuracy. We thus propose the first analysis of RB from the perspective of optimization, which by interpreting RB as a Randomized Block Coordinate Descent in the infinite-dimensional space, gives a faster convergence rate compared to that of other random features. In particular, we show that by drawing R random grids with at least κ number of non-empty bins per grid in expectation, RB method achieves a convergence rate of O(1/κ R)), which not only sharpens its O(1/√R) rate from Monte Carlo analysis, but also shows a κ times speedup over other random features under the same analysis framework. In addition, we demonstrate another advantage of RB in the L1-regularized setting, where unlike other random features, a RB-based Coordinate Descent solver can be parallelized with guaranteed speedup proportional to κ. Our extensive experiments demonstrate the superior performance of the RB features over other random features and kernel approximation methods.

Optimal Convergence for Agnostic Kernel Learning with Random Features.

Optimal Convergence Rates for Agnostic Nyström Kernel Learning.

Optimal Kernel Quantile Learning with Random Features

Learning Adaptive Random Features

Optimal Rates for Agnostic Distributed Learning

Towards Theoretical Understanding of Learning Large-scale Dependent Data Via Random Features

Revisiting Random Binning Feature: Fast Convergence and Strong Parallelizability

A Unified Framework of Random Feature KLMS Algorithms and Convergence Analysis

Towards Unbiased Random Features with Lower Variance for Stationary Indefinite Kernels

Variance-Reducing Couplings for Random Features

A Kernel Least Mean Square Algorithm Based on Randomized Feature Networks

Random Features for Shift-Invariant Kernels with Moment Matching

Learning Adaptive Kernels for Statistical Independence Tests.

Improved convergence rates for some kernel random forest algorithms

Ridgeless Regression with Random Features

Optimal Convergence Rates for Distributed Nystro<spacing diaeresis>m Approximation

Decentralized Kernel Ridge Regression Based on Data-Dependent Random Feature

A Double-Variational Bayesian Framework in Random Fourier Features for Indefinite Kernels

A duality framework for analyzing random feature and two-layer neural networks

Error Bounds for Learning with Vector-Valued Random Features