Abstract:Bayesian Neural Networks (BNNs) provide principled estimates of model and data uncertainty by encoding parameters as distributions. This makes them key enablers for reliable AI that can be deployed on safety critical edge systems. These systems can be made resource efficient by restricting synapses to two synaptic states $\{-1,+1\}$ and using a memristive in-memory computing (IMC) paradigm. However, BNNs pose an additional challenge -- they require multiple instantiations for ensembling, consuming extra resources in terms of energy and area. In this work, we propose a novel sparsity-aware optimization for Bayesian Binary Neural Network (BBNN) accelerators that exploits the inherent BBNN sampling sparsity -- most of the network is made up of synapses that have a high probability of being fixed at $\pm1$ and require no sampling. The optimization scheme proposed here exploits the sampling sparsity that exists both among layers, i.e only a few layers of the network contain a majority of the probabilistic synapses, as well as the parameters i.e., a tiny fraction of parameters in these layers require sampling, reducing total sampled parameter count further by up to $86\%$. We demonstrate no loss in accuracy or uncertainty quantification performance for a VGGBinaryConnect network on CIFAR-100 dataset mapped on a custom sparsity-aware phase change memory (PCM) based IMC simulator. We also develop a simple drift compensation technique to demonstrate robustness to drift-induced degradation. Finally, we project latency, energy, and area for sparsity-aware BNN implementation in both pipelined and non-pipelined modes. With sparsity-aware implementation, we estimate upto $5.3 \times$ reduction in area and $8.8\times$ reduction in energy compared to a non-sparsity-aware implementation. Our approach also results in $2.9 \times $ more power efficiency compared to the state-of-the-art BNN accelerator.

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Improving energy efficiency and classification accuracy of neuromorphic chips by learning binary synaptic crossbars

Improving classification accuracy of feedforward neural networks for spiking neuromorphic chips

Gradient-descent hardware-aware training and deployment for mixed-signal Neuromorphic processors

Parallel Hyperparameter Optimization Of Spiking Neural Network

Sparsity-Aware Optimization of In-Memory Bayesian Binary Neural Network Accelerators

Hardware Platform-Aware Binarized Neural Network Model Optimization

Towards On-Chip Bayesian Neuromorphic Learning

Hardware-friendly Neural Network Architecture for Neuromorphic Computing

Deep Spiking Binary Neural Network for Digital Neuromorphic Hardware

Layer-wise synapse optimization for implementing neural networks on general neuromorphic architectures

Binary stochasticity enabled highly efficient neuromorphic deep learning achieves better-than-software accuracy

Technology Aware Training in Memristive Neuromorphic Systems based on non-ideal Synaptic Crossbars

Hardware architecture and routing-aware training for optimal memory usage: a case study

Data and Power Efficient Intelligence with Neuromorphic Learning Machines

Scalable NoC-based Neuromorphic Hardware Learning and Inference

End-to-End Implementation of Various Hybrid Neural Networks on a Cross-Paradigm Neuromorphic Chip

Compiling Spiking Neural Networks to Neuromorphic Hardware

Short-reach Optical Communications: A Real-world Task for Neuromorphic Hardware

Energy-Efficient On-Board Radio Resource Management for Satellite Communications via Neuromorphic Computing

Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization