Abstract:In this paper, we address learning problems for high dimensional data. Previously, oblivious random projection based approaches that project high dimensional features onto a random subspace have been used in practice for tackling high-dimensionality challenge in machine learning. Recently, various non-oblivious randomized reduction methods have been developed and deployed for solving many numerical problems such as matrix product approximation, low-rank matrix approximation, etc. However, they are less explored for the machine learning tasks, e.g., classification. More seriously, the theoretical analysis of excess risk bounds for risk minimization, an important measure of generalization performance, has not been established for non-oblivious randomized reduction methods. It therefore remains an open problem what is the benefit of using them over previous oblivious random projection based approaches. To tackle these challenges, we propose an algorithmic framework for employing non-oblivious randomized reduction method for general empirical risk minimizing in machine learning tasks, where the original high-dimensional features are projected onto a random subspace that is derived from the data with a small matrix approximation error. We then derive the first excess risk bound for the proposed non-oblivious randomized reduction approach without requiring strong assumptions on the training data. The established excess risk bound exhibits that the proposed approach provides much better generalization performance and it also sheds more insights about different randomized reduction approaches. Finally, we conduct extensive experiments on both synthetic and real-world benchmark datasets, whose dimension scales to O(10^7), to demonstrate the efficacy of our proposed approach.

Online Local Fisher Risk Minimization: a New Online Kernel Method for Online Classification

A Local Online Learning Approach for Non-linear Data.

Model-Based Online Learning with Kernels.

Online Kernel Learning with a Near Optimal Sparsity Bound

A New Locally Linear KNN Method with an Improved Marginal Fisher Analysis for Image Classification

Towards the Online Learning with Kernels in Classification and Regression.

Adaptive Quasiconformal Kernel Fisher Discriminant Analysis Via Weighted Maximum Margin Criterion

Large Scale Online Kernel Classification

Budget Online Multiple Kernel Learning

Efficient Methods for Online Multiclass Logistic Regression

Nonlinear Online Classification Algorithm with Probability Margin

Fisher discrimination based low rank matrix recovery for face recognition

An efficient renovation on kernel Fisher discriminant analysis and face recognition experiments

One-pass online learning: A local approach

A novel risk-control model for the online portfolio selection of high-frequency transactions

Efficient Non-Oblivious Randomized Reduction for Risk Minimization with Improved Excess Risk Guarantee

A Self-Representation Method with Local Similarity Preserving for Fast Multi-View Outlier Detection

Robust online active learning

A Rank-One Update Algorithm for Fast Solving Kernel Foley–Sammon Optimal Discriminant Vectors

Multikernel Correntropy Based Robust Least Squares One-Class Support Vector Machine

Low-rank kernel regression with preserved locality for multi-class analysis