Abstract:We propose a Bayesian neural network-based continual learning algorithm using Variational Inference, aiming to overcome several drawbacks of existing methods. Specifically, in continual learning scenarios, storing network parameters at each step to retain knowledge poses challenges. This is compounded by the crucial need to mitigate catastrophic forgetting, particularly given the limited access to past datasets, which complicates maintaining correspondence between network parameters and datasets across all sessions. Current methods using Variational Inference with KL divergence risk catastrophic forgetting during uncertain node updates and coupled disruptions in certain nodes. To address these challenges, we propose the following strategies. To reduce the storage of the dense layer parameters, we propose a parameter distribution learning method that significantly reduces the storage requirements. In the continual learning framework employing variational inference, our study introduces a regularization term that specifically targets the dynamics and population of the mean and variance of the parameters. This term aims to retain the benefits of KL divergence while addressing related challenges. To ensure proper correspondence between network parameters and the data, our method introduces an importance-weighted Evidence Lower Bound term to capture data and parameter correlations. This enables storage of common and distinctive parameter hyperspace bases. The proposed method partitions the parameter space into common and distinctive subspaces, with conditions for effective backward and forward knowledge transfer, elucidating the network-parameter dataset correspondence. The experimental results demonstrate the effectiveness of our method across diverse datasets and various combinations of sequential datasets, yielding superior performance compared to existing approaches.

Regularizing Explanations in Bayesian Convolutional Neural Networks

On Regularization for Explaining Graph Neural Networks: An Information Theory Perspective

Explicitly Bayesian Regularizations in Deep Learning

Deep Network Regularization via Bayesian Inference of Synaptic Connectivity

Function-Space Regularization in Neural Networks: A Probabilistic Perspective

BayesNAM: Leveraging Inconsistency for Reliable Explanations

Learning local discrete features in explainable-by-design convolutional neural networks

Learning From Brains How to Regularize Machines

Where to model the epistemic uncertainty of Bayesian convolutional neural networks for classification

Towards Robust Visual Explanations for Deep Convolutional Networks with Weight-Wise Perturbations

A Bayesian convolutional neural network-based generalized linear model

Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs

Robust Counterfactual Explanations for Neural Networks With Probabilistic Guarantees

Extracting Explanations, Justification, and Uncertainty from Black-Box Deep Neural Networks

Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods

Towards Modeling Uncertainties of Self-explaining Neural Networks via Conformal Prediction

Revised Regularization for Efficient Continual Learning through Correlation-Based Parameter Update in Bayesian Neural Networks

Counterfactual explanation of Bayesian model uncertainty

Sparsifying Bayesian neural networks with latent binary variables and normalizing flows

Posterior Regularized Bayesian Neural Network Incorporating Soft and Hard Knowledge Constraints

Explaining Deep Convolutional Neural Networks for Image Classification by Evolving Local Interpretable Model-agnostic Explanations