Abstract:Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.

Discriminative Recurrent Sparse Auto-Encoders

Representation Learning Via a Semi-Supervised Stacked Distance Autoencoder for Image Classification

Stacked Convolutional Sparse Auto-Encoders for Representation Learning

k-Sparse Autoencoders

Learning Sparse Representation Via a Nonlinear Shrinkage Encoder and a Linear Sparse Decoder.

Discriminative Representation Learning with Supervised Auto-encoder

On the convergence of group-sparse autoencoders

Stacked What-Where Auto-encoders

Feature Ensemble Learning Based on Sparse Autoencoders for Image Classification

Rank Ordered Autoencoders

HSAE: A Hessian Regularized Sparse Auto-Encoders.

Sparse, Geometric Autoencoder Models of V1

Sparse Representation Learning of Data by Autoencoders with L_1/2 Regularization

Switched linear encoding with rectified linear autoencoders

Sparse Coding For Classification Via Discrimination Ensemble

Sparse Coding with Multi-Layer Decoders using Variance Regularization

Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

Hierarchical Discriminative Sparse Coding Via Bidirectional Connections

Unsupervised learning with sparse space-and-time autoencoders

Learning Deep Representations Using Convolutional Auto-Encoders with Symmetric Skip Connections

A Semi-Supervised Predictive Sparse Decomposition Based on Task-Driven Dictionary Learning.