Abstract:Linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data, due to their simple geometric interpretations and typically attractive computational properties. These methods capture many data features of interest, such as covariance, dynamical structure, correlation between data sets, input-output relationships, and margin between data classes. Methods have been developed with a variety of names and motivations in many fields, and perhaps as a result the connections between all these methods have not been highlighted. Here we survey methods from this disparate literature as optimization programs over matrix manifolds. We discuss principal component analysis, factor analysis, linear multidimensional scaling, Fisher's linear discriminant analysis, canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, distance metric learning, and more. This optimization framework gives insight to some rarely discussed shortcomings of well-known methods, such as the suboptimality of certain eigenvector solutions. Modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver, which accepts as input data and an objective to be optimized, and returns, as output, an optimal low-dimensional projection of the data. This simple optimization framework further allows straightforward generalizations and novel variants of classical methods, which we demonstrate here by creating an orthogonal-projection canonical correlations analysis. More broadly, this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox, objective-agnostic numerical technology.

Advance of the Sufficient Dimension Reduction

Computational and Theoretical Analysis of Supervised Dimensionality Reduction

Deep Nonlinear Sufficient Dimension Reduction

A Brief Survey of Dimension Reduction

Recent Advances In Supervised Dimension Reduction: A Survey

A Slicing‐Free Perspective to Sufficient Dimension Reduction: Selective Review and Recent Developments

Slicing-free Inverse Regression in High-dimensional Sufficient Dimension Reduction

A new sufficient dimension reduction method via rank divergence

On partial sufficient dimension reduction with applications to partially linear multi-index models

Real-valued Multivariate Dimension Reduction: A Survey

Stable Local Dimensionality Reduction Approaches

Linear Dimensionality Reduction: Survey, Insights, and Generalizations

Sufficient Dimension Reduction Using Hilbert-Schmidt Independence Criterion

An Estimating Equation Approach to Dimension Reduction for Longitudinal Data.

Belted and Ensembled Neural Network for Linear and Nonlinear Sufficient Dimension Reduction

A Unified Framework for Semi-Supervised Dimensionality Reduction

Incremental Regularized Least Squares for Dimensionality Reduction of Large-Scale Data.

A selective review of sufficient dimension reduction for multivariate response regression

Supervised Linear Dimension-Reduction Methods: Review, Extensions, and Comparisons

A dimension reduction based approach for estimation and variable selection in partially linear single-index models with high-dimensional covariates

Learning Heterogeneity in Causal Inference Using Sufficient Dimension Reduction