Abstract:Linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data, due to their simple geometric interpretations and typically attractive computational properties. These methods capture many data features of interest, such as covariance, dynamical structure, correlation between data sets, input-output relationships, and margin between data classes. Methods have been developed with a variety of names and motivations in many fields, and perhaps as a result the connections between all these methods have not been highlighted. Here we survey methods from this disparate literature as optimization programs over matrix manifolds. We discuss principal component analysis, factor analysis, linear multidimensional scaling, Fisher's linear discriminant analysis, canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, distance metric learning, and more. This optimization framework gives insight to some rarely discussed shortcomings of well-known methods, such as the suboptimality of certain eigenvector solutions. Modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver, which accepts as input data and an objective to be optimized, and returns, as output, an optimal low-dimensional projection of the data. This simple optimization framework further allows straightforward generalizations and novel variants of classical methods, which we demonstrate here by creating an orthogonal-projection canonical correlations analysis. More broadly, this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox, objective-agnostic numerical technology.

Overview of High Dimensional Linear Regression Models

Computational and Theoretical Analysis of Supervised Dimensionality Reduction

Partially Functional Linear Regression in High Dimensions

Robust linear regression for high‐dimensional data: An overview

Research on linear regression algorithm

Linear Dimensionality Reduction: Survey, Insights, and Generalizations

High-dimensional analysis of variance in multivariate linear regression

Model Diagnosis for Parametric Regression in High-Dimensional Spaces

Estimation of Linear Functionals in High-Dimensional Linear Models: From Sparsity to Nonsparsity

On The Equivalent Of Low-Rank Regressions And Linear Discriminant Analysis Based Regressions

Interpretation of High-Dimensional Linear Regression: Effects of Nullspace and Regularization Demonstrated on Battery Data

High-dimensional Factor Model and Its Applications to Statistical Machine Learning

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

High-Dimensional Analysis for Generalized Nonlinear Regression: from Asymptotics to Algorithm

Advanced Statistics: Linear Regression, Part II: Multiple Linear Regression

Asymptotics of Linear Regression with Linearly Dependent Data

Linear Hypothesis Testing in Linear Models With High-Dimensional Responses

Testing a Single Regression Coefficient in High Dimensional Regression Model

A Review On Dimension Reduction

A Selective Overview of Variable Selection in High Dimensional Feature Space (Invited Review Article)

Supervised Linear Dimension-Reduction Methods: Review, Extensions, and Comparisons