Abstract:With recent advancing of Internet of Things (IoTs), it becomes very attractive to implement the deep convolutional neural networks (DCNNs) onto embedded/portable systems. Presently, executing the software-based DCNNs requires high-performance server clusters in practice, restricting their widespread deployment on the mobile devices. To overcome this issue, considerable research efforts have been conducted in the context of developing highly-parallel and specific DCNN hardware, utilizing GPGPUs, FPGAs, and ASICs. Stochastic Computing (SC), which uses bit-stream to represent a number within [-1, 1] by counting the number of ones in the bit-stream, has a high potential for implementing DCNNs with high scalability and ultra-low hardware footprint. Since multiplications and additions can be calculated using AND gates and multiplexers in SC, significant reductions in power/energy and hardware footprint can be achieved compared to the conventional binary arithmetic implementations. The tremendous savings in power (energy) and hardware resources bring about immense design space for enhancing scalability and robustness for hardware DCNNs. This paper presents the first comprehensive design and optimization framework of SC-based DCNNs (SC-DCNNs). We first present the optimal designs of function blocks that perform the basic operations, i.e., inner product, pooling, and activation function. Then we propose the optimal design of four types of combinations of basic function blocks, named feature extraction blocks, which are in charge of extracting features from input feature maps. Besides, weight storage methods are investigated to reduce the area and power/energy consumption for storing weights. Finally, the whole SC-DCNN implementation is optimized, with feature extraction blocks carefully selected, to minimize area and power/energy consumption while maintaining a high network accuracy level.

Optimization of Softmax Layer in Deep Neural Network Using Integral Stochastic Computation

Efficient Hardware Architecture of Softmax Layer in Deep Neural Network

Efficient Softmax Hardware Architecture for Deep Neural Networks

Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks.

DSCNN: Hardware-oriented Optimization for Stochastic Computing Based Deep Convolutional Neural Networks.

Towards Budget-Driven Hardware Optimization for Deep Convolutional Neural Networks Using Stochastic Computing

Softmax Optimizations for Intel Xeon Processor-based Platforms

Towards Acceleration of Deep Convolutional Neural Networks Using Stochastic Computing

A Low-Cost Implementation Method on Deep Neural Network Using Stochastic Computing

Structural design optimization for deep convolutional neural networks using stochastic computing.

Hardware-Aware Softmax Approximation for Deep Neural Networks

Efficient Precision-Adjustable Architecture for Softmax Function in Deep Learning

A High-Speed and Low-Complexity Architecture for Softmax Function in Deep Learning

Hardware-driven Nonlinear Activation for Stochastic Computing Based Deep Convolutional Neural Networks.

High Performance and Hardware Efficient Stochastic Computing Elements for Deep Neural Network

Fully-Parallel Area-Efficient Deep Neural Network Design Using Stochastic Computing

Hardware-Efficient SoftMax Architecture With Bit-Wise Exponentiation and Reciprocal Calculation

SC-DCNN: Highly-Scalable Deep Convolutional Neural Network using Stochastic Computing

Efficient FPGA Implementation of softmax Layer in Deep Neural Network

Improving Deep Neural Network Performance by Integrating Kernelized Min-Max Objective

High-Precision Method and Architecture for Base-2 Softmax Function in DNN Training.