Abstract:Dimension reduction algorithms are a crucial part of many data science pipelines, including data exploration, feature creation and selection, and denoising. Despite their wide utilization, many non-linear dimension reduction algorithms are poorly understood from a theoretical perspective. In this work we consider a generalized version of multidimensional scaling, which is posed as an optimization problem in which a mapping from a high-dimensional feature space to a lower-dimensional embedding space seeks to preserve either inner products or norms of the distribution in feature space, and which encompasses many commonly used dimension reduction algorithms. We analytically investigate the variational properties of this problem, leading to the following insights: 1) Solutions found using standard particle descent methods may lead to non-deterministic embeddings, 2) A relaxed or probabilistic formulation of the problem admits solutions with easily interpretable necessary conditions, 3) The globally optimal solutions to the relaxed problem actually must give a deterministic embedding. This progression of results mirrors the classical development of optimal transportation, and in a case relating to the Gromov-Wasserstein distance actually gives explicit insight into the structure of the optimal embeddings, which are parametrically determined and discontinuous. Finally, we illustrate that a standard computational implementation of this task does not learn deterministic embeddings, which means that it learns sub-optimal mappings, and that the embeddings learned in that context have highly misleading clustering structure, underscoring the delicate nature of solving this problem computationally.

On the convergence of maximum variance unfolding

Relaxed Maximum-Variance Unfolding

Developments of Two Supervised Maximum Variance Unfolding Algorithms for Process Classification

A Linear Maximum Variance Unfolding Algorithm and Its Application in Image Recognition

On consistent estimation of dimension values

Learning Isometric Separation Maps

Stable Local Dimensionality Reduction Approaches

Convergence Rates of Spectral Distributions of Large Sample Covariance Matrices

Multi-manifold learning based on kernel matrix learning

Convergence and Rate of Convergence of a Manifold-Based Dimension Reduction Algorithm

Learning High Dimensional Correspondences Based on Maximum Variance Unfolding

Likelihood-based Dimension Folding on Tensor Data

Convex optimization learning of faithful Euclidean distance representations in nonlinear dimensionality reduction

On Extreme Value Asymptotics of Projected Sample Covariances in High Dimensions with Applications in Finance and Convolutional Networks

Convergence of coordinate ascent variational inference for log-concave measures via optimal transport

Maximum likelihood inference for high-dimensional problems with multiaffine variable relations

On Probabilistic Embeddings in Optimal Dimension Reduction

Convergence Rates of Eigenvector Empirical Spectral Distribution of Large Dimensional Sample Covariance Matrix

Fast Convergence of Langevin Dynamics on Manifold: Geodesics meet Log-Sobolev

Unified Framework for Spectral Dimensionality Reduction, Maximum Variance Unfolding, and Kernel Learning By Semidefinite Programming: Tutorial and Survey

Convergence of Unadjusted Langevin in High Dimensions: Delocalization of Bias