Abstract:In Bourlard and Kamp (Biol Cybern 59(4):291-294, 1998), it was theoretically proven that autoencoders (AE) with single hidden layer (previously called "auto-associative multilayer perceptrons") were, in the best case, implementing singular value decomposition (SVD) Golub and Reinsch (Linear algebra, Singular value decomposition and least squares solutions, pp 134-151. Springer, 1971), equivalent to principal component analysis (PCA) Hotelling (Educ Psychol 24(6/7):417-441, 1993); Jolliffe (Principal component analysis, springer series in statistics, 2nd edn. Springer, New York ). That is, AE are able to derive the eigenvalues that represent the amount of variance covered by each component even with the presence of the nonlinear function (sigmoid-like, or any other nonlinear functions) present on their hidden units. Today, with the renewed interest in "deep neural networks" (DNN), multiple types of (deep) AE are being investigated as an alternative to manifold learning Cayton (Univ California San Diego Tech Rep 12(1-17):1, 2005) for conducting nonlinear feature extraction or fusion, each with its own specific (expected) properties. Many of those AE are currently being developed as powerful, nonlinear encoder-decoder models, or used to generate reduced and discriminant feature sets that are more amenable to different modeling and classification tasks. In this paper, we start by recalling and further clarifying the main conclusions of Bourlard and Kamp (Biol Cybern 59(4):291-294, 1998), supporting them by extensive empirical evidences, which were not possible to be provided previously (in 1988), due to the dataset and processing limitations. Upon full understanding of the underlying mechanisms, we show that it remains hard (although feasible) to go beyond the state-of-the-art PCA/SVD techniques for auto-association. Finally, we present a brief overview on different autoencoder models that are mainly in use today and discuss their rationale, relations and application areas.

Exponential Capacity in an Autoencoder Neural Network with a Hidden Layer

Why should autoencoders work?

Exponential Capacity of Dense Associative Memories

The Exponential Capacity of Dense Associative Memories

A NEW TRAINING ALGORITHM OF AUTOENCODER COMBINING LATERAL INHIBITION MECHANISM

A Capacity Scaling Law for Artificial Neural Networks

Autoencoders reloaded

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck

Binary autoencoder with random binary weights

Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning

Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning

On measuring excess capacity in neural networks

Capacity of the covariance perceptron

Deep Neural Network Capacity

Encoding innate ability through a genomic bottleneck

Do autoencoders need a bottleneck for anomaly detection?

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions

The Persian Rug: solving toy models of superposition using large-scale symmetries

Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders