Abstract:Binary Neural Networks (BNNs) are showing tremendous success on realistic image classification tasks. Notably, their accuracy is similar to the state-of-the-art accuracy obtained by full-precision models tailored to edge devices. In this regard, BNNs are very amenable to edge devices since they employ 1-bit to store the inputs and weights, and thus, their storage requirements are low. Also, BNNs computations are mainly done using xnor and pop-counts operations which are implemented very efficiently using simple hardware structures. Nonetheless, supporting BNNs efficiently on mobile CPUs is far from trivial since their benefits are hindered by frequent memory accesses to load weights and inputs. In BNNs, a weight or an input is stored using one bit, and aiming to increase storage and computation efficiency, several of them are packed together as a sequence of bits. In this work, we observe that the number of unique sequences representing a set of weights is typically low. Also, we have seen that during the evaluation of a BNN layer, a small group of unique sequences is employed more frequently than others. Accordingly, we propose exploiting this observation by using Huffman Encoding to encode the bit sequences and then using an indirection table to decode them during the BNN evaluation. Also, we propose a clustering scheme to identify the most common sequences of bits and replace the less common ones with some similar common sequences. Hence, we decrease the storage requirements and memory accesses since common sequences are encoded with fewer bits. We extend a mobile CPU by adding a small hardware structure that can efficiently cache and decode the compressed sequence of bits. We evaluate our scheme using the ReAacNet model with the Imagenet dataset. Our experimental results show that our technique can reduce memory requirement by 1.32x and improve performance by 1.35x.

An efficient GPU-accelerated inference engine for binary neural network on mobile phones

PhoneBit: Efficient GPU-Accelerated Binary Neural Network Inference Engine for Mobile Phones

Explore Training of Deep Convolutional Neural Networks on Battery-powered Mobile Devices: Design and Application

Close the Gap Between Deep Learning and Mobile Intelligence by Incorporating Training in the Loop

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Binary Neural Networks as a general-propose compute paradigm for on-device computer vision

On-Device Neural Net Inference with Mobile GPUs

SparseByteNN: A Novel Mobile Inference Acceleration Framework Based on Fine-Grained Group Sparsity

BitFlow: Exploiting Vector Parallelism for Binary Neural Networks on CPU

BitGNN: Unleashing the Performance Potential of Binary Graph Neural Networks on GPUs

Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks

Exploiting Kernel Compression on BNNs

MNN: A Universal and Efficient Inference Engine

Profiling and optimizing deep learning inference on mobile GPUs.

Optimizing data-flow in Binary Neural Networks

Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs

26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone

An Approach of Binary Neural Network Energy-Efficient Implementation

Mobile or FPGA? A Comprehensive Evaluation on Energy Efficiency and a Unified Optimization Framework

A Power Efficient Neural Network Implementation on Heterogeneous FPGA and GPU Devices

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing