Abstract:Bayesian Neural Networks (BayNNs) naturally provide uncertainty in their predictions, making them a suitable choice in safety-critical applications. Additionally, their realization using memristor-based in-memory computing (IMC) architectures enables them for resource-constrained edge applications. In addition to predictive uncertainty, however, the ability to be inherently robust to noise in computation is also essential to ensure functional safety. In particular, memristor-based IMCs are susceptible to various sources of non-idealities such as manufacturing and runtime variations, drift, and failure, which can significantly reduce inference accuracy. In this paper, we propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in IMC architectures. To achieve this, we introduce a novel normalization layer combined with stochastic affine transformations. Empirical results in various benchmark datasets show a graceful degradation in inference accuracy, with an improvement of up to $58.11\%$.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to enhance the reliability and inference accuracy of Bayesian neural networks (BayNNs) in memristor - based in - memory computing (IMC) architectures in the edge - computing environment. Specifically, the authors focus on: 1. **Improving robustness to noise**: Memristor - based IMC architectures are susceptible to non - ideal factors such as manufacturing and runtime variations, drifts, and faults, which can significantly reduce inference accuracy. Therefore, a method is needed to make BayNNs more robust in this environment. 2. **Maintaining the ability to estimate uncertainty**: In addition to improving inference accuracy, it is also necessary to ensure that BayNNs can continue to provide prediction uncertainty estimates, which are crucial for safety - critical applications such as autonomous driving and medical diagnosis. 3. **Without explicit non - ideal modeling**: Existing methods usually require non - ideal modeling and training for each NVM technology, which not only increases complexity but also limits their generality. Therefore, a method that does not require explicit modeling is needed. 4. **Applicable to different NVM technologies and non - idealities**: The proposed method should be widely applicable to different NVM technologies and non - idealities, not just improvements in specific cases. To solve the above problems, the authors propose an inverted normalization layer with stochastic affine transformations. Through this method, the authors aim to: - Improve the inference accuracy and robustness of BayNNs in IMC architectures. - Maintain the ability of BayNNs to estimate uncertainty. - Simplify the training process to make it easier to implement. - Make the method more general and applicable to different types of NVM technologies and non - idealities. Experimental results show that this method performs well on multiple benchmark datasets, with a small decrease in inference accuracy and, in some cases, a significant improvement compared to existing methods. In addition, this method also shows higher robustness in the case of bit - flip faults and conductivity changes. ### Formula presentation Some of the formulas involved in the paper are as follows: 1. **Batch Normalization**: \[ \hat{y}=\frac{y - \mu}{\sqrt{\sigma^{2}+\epsilon}} \] where $\mu$ and $\sigma$ are the batch mean and standard deviation respectively, and $\epsilon$ is a small constant used for numerical stability. 2. **Affine Transformation**: \[ y'=\gamma\hat{y}+\beta \] where $\gamma$ and $\beta$ are learnable parameters. 3. **Inverted Normalization**: In inverted normalization, affine transformation is performed before normalization: \[ y' = \text{normalize}(\gamma y+\beta) \] 4. **Stochastic Affine Dropout**: Stochastic affine Dropout introduces randomness by randomly dropping weights and biases: \[ y'=\text{normalize}(w\odot M_{w}\cdot y + b\odot M_{b}) \] where $M_{w}$ and $M_{b}$ are binary masks sampled from the Bernoulli distribution, used for weights and biases respectively. Through these innovations, the authors have successfully improved the reliability and accuracy of BayNNs in IMC architectures while maintaining their ability to estimate uncertainty.

Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations

Reliable Memristor-based Neuromorphic Design Using Variation- and Defect-Aware Training

Temporal Effective Batch Normalization in Spiking Neural Networks

Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning

Few-Shot Testing: Estimating Uncertainty of Memristive Deep Neural Networks Using One Bayesian Test Vector

Sparsity-Aware Optimization of In-Memory Bayesian Binary Neural Network Accelerators

Enabling uncertainty estimation in neural networks through weight perturbation for improved Alzheimer's disease classification

Two-dimensional materials-based probabilistic synapses and reconfigurable neurons for measuring inference uncertainty using Bayesian neural networks

Efficient and Mathematically Robust Operations for Certified Neural Networks Inference

SNN-RAT: Robustness-enhanced Spiking Neural Network through Regularized Adversarial Training

Optimizing Artificial Neural Networks to Minimize Arithmetic Errors in Stochastic Computing Implementations

Neural Sampling Machine with Stochastic Synapse allows Brain-like Learning and Inference

NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes

Iterative Normalization: Beyond Standardization towards Efficient Whitening

Negative Feedback Training: A Novel Concept to Improve Robustness of NVCIM DNN Accelerators

Examining the Role and Limits of Batchnorm Optimization to Mitigate Diverse Hardware-noise in In-memory Computing

BayesNetCNN: incorporating uncertainty in neural networks for image-based classification tasks

Towards Regression-Free Neural Networks for Diverse Compute Platforms

Measuring and Mitigating Local Instability in Deep Neural Networks

BayesFT: Bayesian Optimization for Fault Tolerant Neural Network Architecture

Improving Neural Additive Models with Bayesian Principles