Abstract:Deep convolutional neural networks (DCNNs) are one of the most promising deep learning techniques and have been recognized as the dominant approach for almost all recognition and detection tasks. The computation of DCNNs is memory intensive due to large feature maps and neuron connections, and the performance highly depends on the capability of hardware resources. With the recent trend of wearable devices and Internet of Things, it becomes desirable to integrate the DCNNs onto embedded and portable devices that require low power and energy consumptions and small hardware footprints. Recently stochastic computing (SC)-DCNN demonstrated that SC as a low-cost substitute to binary-based computing radically simplifies the hardware implementation of arithmetic units and has the potential to satisfy the stringent power requirements in embedded devices. In SC, many arithmetic operations that are resource-consuming in binary designs can be implemented with very simple hardware logic, alleviating the extensive computational complexity. It offers a colossal design space for integration and optimization due to its reduced area and soft error resiliency. In this paper, we present HEIF, a highly efficient SC-based inference framework of the large-scale DCNNs, with broad applications including (but not limited to) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LeNet-5 and AlexNet</italic> , that achieves high energy efficiency and low area/hardware cost. Compared to SC-DCNN, HEIF features: 1) the first (to the best of our knowledge) SC-based rectified linear unit activation function to catch up with the recent advances in software models and mitigate degradation in application-level accuracy; 2) the redesigned approximate parallel counter and optimized stochastic multiplication using transmission gates and inverse mirror adders; and 3) the new optimization of weight storage using clustering. Most importantly, to achieve maximum energy efficiency while maintaining acceptable accuracy, HEIF considers holistic optimizations on cascade connection of function blocks in DCNN, pipelining technique, and bit-stream length reduction. Experimental results show that in large-scale applications HEIF outperforms previous SC-DCNN by the throughput of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.1\times $ </tex-math></inline-formula> , by area efficiency of up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$6.5\times $ </tex-math></inline-formula> , and achieves up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${5.6\times }$ </tex-math></inline-formula> energy improvement.

Structural design optimization for deep convolutional neural networks using stochastic computing.

DSCNN: Hardware-oriented Optimization for Stochastic Computing Based Deep Convolutional Neural Networks.

SC-DCNN: Highly-Scalable Deep Convolutional Neural Network using Stochastic Computing

Towards Acceleration of Deep Convolutional Neural Networks Using Stochastic Computing

Towards Budget-Driven Hardware Optimization for Deep Convolutional Neural Networks Using Stochastic Computing

Stochastic Computing Hardware Design and Optimization for Convolutional Neutral Networks

Normalization and Dropout for Stochastic Computing-Based Deep Convolutional Neural Networks.

Fully-Parallel Area-Efficient Deep Neural Network Design Using Stochastic Computing

Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks.

Optimization for Deep Convolutional Neural Network of Stochastic Computing on MLC-PCM-based System

Hardware-driven Nonlinear Activation for Stochastic Computing Based Deep Convolutional Neural Networks.

An area and energy efficient design of domain-wall memory-based deep convolutional neural networks using stochastic computing

Optimizing Stochastic Computing for Low Latency Inference of Convolutional Neural Networks

HEIF: Highly Efficient Stochastic Computing based Inference Framework for Deep Neural Networks

Hardware-aware Neural Architecture Search for Stochastic Computing-Based Neural Networks on Tiny Devices

A Survey of Stochastic Computing Neural Networks for Machine Learning Applications

Efficient Fast Convolution Architecture Based on Stochastic Computing

Designing Reconfigurable Large-Scale Deep Learning Systems Using Stochastic Computing

Hybrid Stochastic-Binary Computing for Low-Latency and High-Precision Inference of CNNs

Stochastic Computing Convolution Neural Network Architecture Reinvented For Highly Efficient Artificial Intelligence Workload on Field Programmable Gate Array

Efficient Compression Methods for Wire-Spread-Based Stochastic Computing Deep Neural Networks