Abstract:In Bourlard and Kamp (Biol Cybern 59(4):291-294, 1998), it was theoretically proven that autoencoders (AE) with single hidden layer (previously called "auto-associative multilayer perceptrons") were, in the best case, implementing singular value decomposition (SVD) Golub and Reinsch (Linear algebra, Singular value decomposition and least squares solutions, pp 134-151. Springer, 1971), equivalent to principal component analysis (PCA) Hotelling (Educ Psychol 24(6/7):417-441, 1993); Jolliffe (Principal component analysis, springer series in statistics, 2nd edn. Springer, New York ). That is, AE are able to derive the eigenvalues that represent the amount of variance covered by each component even with the presence of the nonlinear function (sigmoid-like, or any other nonlinear functions) present on their hidden units. Today, with the renewed interest in "deep neural networks" (DNN), multiple types of (deep) AE are being investigated as an alternative to manifold learning Cayton (Univ California San Diego Tech Rep 12(1-17):1, 2005) for conducting nonlinear feature extraction or fusion, each with its own specific (expected) properties. Many of those AE are currently being developed as powerful, nonlinear encoder-decoder models, or used to generate reduced and discriminant feature sets that are more amenable to different modeling and classification tasks. In this paper, we start by recalling and further clarifying the main conclusions of Bourlard and Kamp (Biol Cybern 59(4):291-294, 1998), supporting them by extensive empirical evidences, which were not possible to be provided previously (in 1988), due to the dataset and processing limitations. Upon full understanding of the underlying mechanisms, we show that it remains hard (although feasible) to go beyond the state-of-the-art PCA/SVD techniques for auto-association. Finally, we present a brief overview on different autoencoder models that are mainly in use today and discuss their rationale, relations and application areas.

Transforming Auto-Encoders

Auto-Encoding Transformations in Reparameterized Lie Groups for Unsupervised Learning.

Neural Isometries: Taming Transformations for Equivariant ML

Extreme Image Transformations Affect Humans and Machines Differently

Masked Autoencoders Are Scalable Vision Learners

Autoencoders reloaded

The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision

Auto-Encoding for Shared Cross Domain Feature Representation and Image-to-Image Translation

Denoising Auto-Encoders Toward Robust Unsupervised Feature Representation.

Autoencoding Features for Aviation Machine Learning Problems

Depth and Representation in Vision Models

Why should autoencoders work?

Video-Specific Autoencoders for Exploring, Editing and Transmitting Videos

Training Invertible Neural Networks as Autoencoders

Transferable polychromatic optical encoder for neural networks

Computer Vision : History , the Rise of Deep Networks , and Future Vistas Panel on Perception and Cognition , MORS Meeting on Artificial Intelligence and Autonomy

EncodeNet: A Framework for Boosting DNN Accuracy with Entropy-driven Generalized Converting Autoencoder

pAE: An Efficient Autoencoder Architecture for Modeling the Lateral Geniculate Nucleus by Integrating Feedforward and Feedback Streams in Human Visual System

Stacked Auto-Encoders for Feature Extraction with Neural Networks.

About the Application of Autoencoders For Visual Defect Detection

Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”