Abstract:Deep neural networks are state-of-the-art models for understanding the content of images, video and raw input data. However, implementing a deep neural network in embedded systems is a challenging task, because a typical deep neural network, such as a Deep Belief Network using 128x128 images as input, could exhaust Giga bytes of memory and result in bandwidth and computing bottleneck. To address this challenge, this paper presents a hardware-oriented deep learning algorithm, named as the Deep Adaptive Network, which attempts to exploit the sparsity in the neural connections. The proposed method adaptively reduces the weights associated with negligible features to zero, leading to sparse feedforward network architecture. Furthermore, since the small proportion of important weights are significantly larger than zero, they can be robustly thresholded and represented using single-bit integers (-1 and +1), leading to implementations of deep neural networks with sparse and binary connections. Our experiments showed that, for the application of recognizing MNIST handwritten digits, the features extracted by a two-layer Deep Adaptive Network with about 25% reserved important connections achieved 97.2% classification accuracy, which was almost the same with the standard Deep Belief Network (97.3%). Furthermore, for efficient hardware implementations, the sparse-and-binary-weighted deep neural network could save about 99.3% memory and 99.9% computation units without significant loss of classification accuracy for pattern recognition applications.

Sparse Deep Belief Net For Handwritten Digits Classification

Training Deep Belief Network with Sparse Hidden Units.

Learning and real-time classification of hand-written digits with spiking neural networks

Learning Sparse Feature Representations using Probabilistic Quadtrees and Deep Belief Nets

Massively Deep Artificial Neural Networks for Handwritten Digit Recognition

NeuroWrite: Predictive Handwritten Digit Classification using Deep Neural Networks

Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections

Deep Adaptive Networks for image classification

An Adaptive Deep Belief Network With Sparse Restricted Boltzmann Machines

Restricted Boltzmann Machine with Adaptive Local Hidden Units.

A Fast Learning Algorithm for Deep Belief Nets

Text Classification Based on Deep Belief Network and Softmax Regression.

BDNet: Bengali Handwritten Numeral Digit Recognition based on Densely connected Convolutional Neural Networks

Sparse Group Restricted Boltzmann Machines

Deep Network with Pixel-Level Rectification and Robust Training for Handwriting Recognition

A regularization-reinforced DBN for digital recognition

Handwritten Digit Recognition of MNIST dataset using Deep Learning state-of-the-art Artificial Neural Network (ANN) and Convolutional Neural Network (CNN)

Deep networks for online handwriting Chinese character recognition

Improved Classification with Semi-supervised Deep Belief Network

Multilayer Self-Organizing Impulse Neural Network for Handwritten Digit Recognition

Discriminative deep belief networks for visual data classification