Abstract:In Bourlard and Kamp (Biol Cybern 59(4):291-294, 1998), it was theoretically proven that autoencoders (AE) with single hidden layer (previously called "auto-associative multilayer perceptrons") were, in the best case, implementing singular value decomposition (SVD) Golub and Reinsch (Linear algebra, Singular value decomposition and least squares solutions, pp 134-151. Springer, 1971), equivalent to principal component analysis (PCA) Hotelling (Educ Psychol 24(6/7):417-441, 1993); Jolliffe (Principal component analysis, springer series in statistics, 2nd edn. Springer, New York ). That is, AE are able to derive the eigenvalues that represent the amount of variance covered by each component even with the presence of the nonlinear function (sigmoid-like, or any other nonlinear functions) present on their hidden units. Today, with the renewed interest in "deep neural networks" (DNN), multiple types of (deep) AE are being investigated as an alternative to manifold learning Cayton (Univ California San Diego Tech Rep 12(1-17):1, 2005) for conducting nonlinear feature extraction or fusion, each with its own specific (expected) properties. Many of those AE are currently being developed as powerful, nonlinear encoder-decoder models, or used to generate reduced and discriminant feature sets that are more amenable to different modeling and classification tasks. In this paper, we start by recalling and further clarifying the main conclusions of Bourlard and Kamp (Biol Cybern 59(4):291-294, 1998), supporting them by extensive empirical evidences, which were not possible to be provided previously (in 1988), due to the dataset and processing limitations. Upon full understanding of the underlying mechanisms, we show that it remains hard (although feasible) to go beyond the state-of-the-art PCA/SVD techniques for auto-association. Finally, we present a brief overview on different autoencoder models that are mainly in use today and discuss their rationale, relations and application areas.

AE2-Nets: Autoencoder in Autoencoder Networks

Auto-AD: Autonomous Hyperspectral Anomaly Detection Network Based on Fully Convolutional Autoencoder

Learning Deep Network Representations with Adversarially Regularized Autoencoders

Hyperspectral Anomaly Detection with Auto-Encoder and Independent Target

Semantic-Aware Auto-Encoders for Self-supervised Representation Learning

A tutorial on multi-view autoencoders using the multi-view-AE library

Multi-View Graph Autoencoder for Unsupervised Graph Representation Learning

Multi-Way Multi-View Deep Autoencoder for Image Feature Learning with Multi-Level Graph Regularization.

AENEA: A novel autoencoder-based network embedding algorithm

Diversified feature representation via deep auto-encoder ensemble through multiple activation functions

Elastic deep multi-view autoencoder with diversity embedding

Representation learning via an integrated autoencoder for unsupervised domain adaptation

Multi-local Collaborative AutoEncoder

L2G Auto-encoder

Autoencoders reloaded

Reconstruction of Hidden Representation for Robust Feature Extraction

Uncertainty-Aware Multi-View Representation Learning

Masked Autoencoders Are Robust Neural Architecture Search Learners

Latent space configuration for improved generalization in supervised autoencoder neural networks

UniM^2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

Knowledge-integrated autoencoder model