Abstract:We explore two primary classes of approaches to dimensionality reduction (DR): Independent Dimensionality Reduction (IDR) and Simultaneous Dimensionality Reduction (SDR). In IDR methods, of which Principal Components Analysis is a paradigmatic example, each modality is compressed independently, striving to retain as much variation within each modality as possible. In contrast, in SDR, one simultaneously compresses the modalities to maximize the covariation between the reduced descriptions while paying less attention to how much individual variation is preserved. Paradigmatic examples include Partial Least Squares and Canonical Correlations Analysis. Even though these DR methods are a staple of statistics, their relative accuracy and data set size requirements are poorly understood. We introduce a generative linear model to synthesize multimodal data with known variance and covariance structures to examine these questions. We assess the accuracy of the reconstruction of the covariance structures as a function of the number of samples, signal-to-noise ratio, and the number of varying and covarying signals in the data. Using numerical experiments, we demonstrate that linear SDR methods consistently outperform linear IDR methods and yield higher-quality, more succinct reduced-dimensional representations with smaller datasets. Remarkably, regularized CCA can identify low-dimensional weak covarying structures even when the number of samples is much smaller than the dimensionality of the data, which is a regime challenging for all dimensionality reduction methods. Our work corroborates and explains previous observations in the literature that SDR can be more effective in detecting covariation patterns in data. These findings suggest that SDR should be preferred to IDR in real-world data analysis when detecting covariation is more important than preserving variation.

itdr: An R package of Integral Transformation Methods to Estimate the SDR Subspaces in Regression

UDRN: Unified Dimensional Reduction Neural Network for Feature Selection and Feature Projection

The R package psvmSDR: A Unified Algorithm for Sufficient Dimension Reduction via Principal Machines

A Principal Square Response Forward Regression Method for Dimension Reduction

New forest-based approaches for sufficient dimension reduction

FUNCTIONAL SUFFICIENT DIMENSION REDUCTION THROUGH AVERAGE FRÉCHET DERIVATIVES

The flare Package for High Dimensional Linear Regression and Precision Matrix Estimation in R

Dimension Reduction for Fréchet Regression

SIHR: An R Package for Statistical Inference in High-dimensional Linear and Logistic Regression Models

Rigid transformations for stabilized lower dimensional space to support subsurface uncertainty quantification and interpretation

Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning

Projective Resampling Functional Sliced Inverse Regression

Recursive Dimension Reduction for Semisupervised Learning

Sparse kernel sufficient dimension reduction

orthoDr: Semiparametric Dimension Reduction via Orthogonality Constrained Optimization

Spectral regression: a regression framework for efficient regularized subspace learning

Subspace-Based Distorted-Rytov Iterative Method for Solving Inverse Scattering Problems

Asymptotic results for nonparametric regression estimators after sufficient dimension reduction estimation

Functional sufficient dimension reduction through information maximization with application to classification

Robust sufficient dimension reduction via α-distance covariance