Abstract:In this paper, we address learning problems for high dimensional data. Previously, oblivious random projection based approaches that project high dimensional features onto a random subspace have been used in practice for tackling high-dimensionality challenge in machine learning. Recently, various non-oblivious randomized reduction methods have been developed and deployed for solving many numerical problems such as matrix product approximation, low-rank matrix approximation, etc. However, they are less explored for the machine learning tasks, e.g., classification. More seriously, the theoretical analysis of excess risk bounds for risk minimization, an important measure of generalization performance, has not been established for non-oblivious randomized reduction methods. It therefore remains an open problem what is the benefit of using them over previous oblivious random projection based approaches. To tackle these challenges, we propose an algorithmic framework for employing non-oblivious randomized reduction method for general empirical risk minimizing in machine learning tasks, where the original high-dimensional features are projected onto a random subspace that is derived from the data with a small matrix approximation error. We then derive the first excess risk bound for the proposed non-oblivious randomized reduction approach without requiring strong assumptions on the training data. The established excess risk bound exhibits that the proposed approach provides much better generalization performance and it also sheds more insights about different randomized reduction approaches. Finally, we conduct extensive experiments on both synthetic and real-world benchmark datasets, whose dimension scales to O(10^7), to demonstrate the efficacy of our proposed approach.

Leveraging Mixed and Incomplete Outcomes Via Reduced-Rank Modeling.

Robust reduced-rank regression

Efficient Estimation of Reduced-Rank Partial Envelope Model in Multivariate Linear Regression

Simultaneous Dimension Reduction and Variable Selection for Multinomial Logistic Regression

Leveraging independence in high-dimensional mixed linear regression

Reduced-Rank Multi-objective Policy Learning and Optimization

Efficient Inference on High-Dimensional Linear Models with Missing Outcomes

Multivariate Spatiotemporal Models with Low Rank Coefficient Matrix

Best Subset Selection in Reduced Rank Regression

On partial sufficient dimension reduction with applications to partially linear multi-index models

Simultaneous Best Subset Selection and Dimension Reduction via Primal-Dual Iterations

Supervised Multivariate Learning with Simultaneous Feature Auto-grouping and Dimension Reduction

Reduced Rank Regression with Possibly Non-Smooth Criterion Functions: an Empirical Likelihood Approach.

Dimensionality reduction by low-rank embedding

Efficient and Doubly-Robust Methods for Variable Selection and Parameter Estimation in Longitudinal Data Analysis

Large-Scale Sparse Regression for Multiple Responses with Applications to UK Biobank

An Efficient Convex Formulation for Reduced-Rank Linear Discriminant Analysis in High Dimensions

Robust Low Rank Approxiamtion Via Inliers Selection

Efficient Non-Oblivious Randomized Reduction for Risk Minimization with Improved Excess Risk Guarantee

LARGE-SCALE MULTIVARIATE SPARSE REGRESSION WITH APPLICATIONS TO UK BIOBANK

Low-Rank Feature Reduction and Sample Selection for Multi-output Regression.