Abstract:Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.

Learning Deep ℓ_0 Encoders

Learning Deep &#8467;<sub>0</sub> Encoders

Learning A Deep ℓ Encoder for Hashing.

Learning Sparse Representation Via a Nonlinear Shrinkage Encoder and a Linear Sparse Decoder.

Fast Learning with Nonconvex L1-2 Regularization.

Learning A Deep Encoder for Hashing

Sparse Auto-encoder with Smoothed l_1 Regularization

Dictionary Learning with ℓ1/2 Regularizer for Sparsity Based on Proximal Operator

A NEW TRAINING ALGORITHM OF AUTOENCODER COMBINING LATERAL INHIBITION MECHANISM

GIST: General Iterative Shrinkage and Thresholding for Non-convex Sparse Learning

Sparse Auto-encoder with Smoothed $$l_1$$l1 Regularization

A Neurodynamic Approach to L0-Constrained Optimization

Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling

Dictionary Learning with &#x2113;<inf>1/2</inf> Regularizer for Sparsity Based on Proximal Operator

Learning Sparse Neural Networks through L0 Regularization

Neural Network for a Class of Sparse Optimization with L0-regularization.

Learning Deep Sparse Regularizers with Applications to Multi-View Clustering and Semi-Supervised Classification

Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding

Pseudoinverse Learning Algorithom for Fast Sparse Autoencoder Training.

HSAE: A Hessian Regularized Sparse Auto-Encoders.

Structured Sparsity Optimization with Non-Convex Surrogates of L2,0-Norm: A Unified Algorithmic Framework