Abstract:Neural networks, specifically deep convolutional neural networks, have achieved unprecedented performance in various computer vision tasks, but the rationale for the computations and structures of successful neural networks is not fully understood. Theories abound for the aptitude of convolutional neural networks for image classification, but less is understood about why such models would be capable of complex visual tasks such as inference and anomaly identification. Here, we propose a sparse coding interpretation of neural networks that have ReLU activation and of convolutional neural networks in particular. In sparse coding, when the model's basis functions are assumed to be orthogonal, the optimal coefficients are given by the soft-threshold function of the basis functions projected onto the input image. In a non-negative variant of sparse coding, the soft-threshold function becomes a ReLU. Here, we derive these solutions via sparse coding with orthogonal-assumed basis functions, then we derive the convolutional neural network forward transformation from a modified non-negative orthogonal sparse coding model with an exponential prior parameter for each sparse coding coefficient. Next, we derive a complete convolutional neural network without normalization and pooling by adding logistic regression to a hierarchical sparse coding model. Finally we motivate potentially more robust forward transformations by maintaining sparse priors in convolutional neural networks as well performing a stronger nonlinear transformation.

A Novel Hebbian Rules Based Method for Computation of Sparse Coding Basis Vectors.

Image Feature Extraction Method Based On Sparse Coding Framework Of Hypergraph

A New Spatial Encoding Method for Sparse Binary Data

Efficient Coding for Natural Images Based on the Sparseness of Neural Coding in V1 Across the Stimuli

Discriminative Sparse Neighbor Coding.

Unsupervised learning of compositional sparse code for natural image representation

Assignment 2 - Deep Learning with Sparse Coding

Hierarchical Sparse Coding from a Bayesian Perspective.

Hebbian-based Mean Shift for Learning the Diverse Shapes of V1 Simple Cell Receptive Fields

Sparse Concept Coding for Visual Analysis

Local Hybrid Coding for Image Classification

New Evidences for Sparse Coding Strategy Employed in Visual Neurons: from the Image Processing and Nonlinear Approximation Viewpoint.

A New Image Retrieval Algorithm Based on Sparse Coding

Sparse coding for image feature extraction and image recognition

Non-negative and Local Sparse Coding Based on L(2)-Norm and Hessian Regularization

A Sparse Coding Interpretation of Neural Networks and Theoretical Implications

Provably Accurate Double-Sparse Coding

Time varying images sparse coding network model based on simple cell in visual cortex

Low-Dimensional Structures: Sparse Coding for Neuronal Activity

A Noise-Invariant Sparse Coding Model for Binary Input

A Learning-Based Framework for Low Bit-Rate Image and Video Coding