Abstract:While many phenomena in physics and engineering are formally high-dimensional, their long-time dynamics often live on a lower-dimensional manifold. The present work introduces an autoencoder framework that combines implicit regularization with internal linear layers and $L_2$ regularization (weight decay) to automatically estimate the underlying dimensionality of a data set, produce an orthogonal manifold coordinate system, and provide the mapping functions between the ambient space and manifold space, allowing for out-of-sample projections. We validate our framework's ability to estimate the manifold dimension for a series of datasets from dynamical systems of varying complexities and compare to other state-of-the-art estimators. We analyze the training dynamics of the network to glean insight into the mechanism of low-rank learning and find that collectively each of the implicit regularizing layers compound the low-rank representation and even self-correct during training. Analysis of gradient descent dynamics for this architecture in the linear case reveals the role of the internal linear layers in leading to faster decay of a "collective weight variable" incorporating all layers, and the role of weight decay in breaking degeneracies and thus driving convergence along directions in which no decay would occur in its absence. We show that this framework can be naturally extended for applications of state-space modeling and forecasting by generating a data-driven dynamic model of a spatiotemporally chaotic partial differential equation using only the manifold coordinates. Finally, we demonstrate that our framework is robust to hyperparameter choices.

Dimensionality Reduction and Reconstruction of Data Based on Autoencoder Network

Deep Manifold Transformation for Dimension Reduction

Auto-Encoder Based Dimensionality Reduction

Dimensionality Reduction Strategy Based on Auto-Encoder

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

Generalized Autoencoder: A Neural Network Framework for Dimensionality Reduction

An Automated Data Mining Framework Using Autoencoders for Feature Extraction and Dimensionality Reduction

Dimension Estimation Using Autoencoders

A restorable autoencoder as a method for dimensionality reduction

A Hybrid Autoencoder Framework of Dimensionality Reduction for Brain-Computer Interface Decoding

Autoencoders with Intrinsic Dimension Constraints for Learning Low Dimensional Image Representations

High Dimensionality Reduction Using Cur Matrix Decomposition And Auto-Encoder For Web Image Classification

SCDRHA: A scRNA-Seq Data Dimensionality Reduction Algorithm Based on Hierarchical Autoencoder

Sparse Clustering Algorithm Based on Multi-Domain Dimensionality Reduction Autoencoder

Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems

An Additive Autoencoder for Dimension Estimation

Supervised Dimensionality Reduction and Image Classification Utilizing Convolutional Autoencoders

Why should autoencoders work?

Guided autoencoder for dimensionality reduction of pedestrian features

A Dictionary-Based Algorithm for Dimensionality Reduction and Data Reconstruction

A Meta-learning Formulation of the Autoencoder Problem for Non-linear Dimensionality Reduction