Abstract:We propose a Bayesian neural network-based continual learning algorithm using Variational Inference, aiming to overcome several drawbacks of existing methods. Specifically, in continual learning scenarios, storing network parameters at each step to retain knowledge poses challenges. This is compounded by the crucial need to mitigate catastrophic forgetting, particularly given the limited access to past datasets, which complicates maintaining correspondence between network parameters and datasets across all sessions. Current methods using Variational Inference with KL divergence risk catastrophic forgetting during uncertain node updates and coupled disruptions in certain nodes. To address these challenges, we propose the following strategies. To reduce the storage of the dense layer parameters, we propose a parameter distribution learning method that significantly reduces the storage requirements. In the continual learning framework employing variational inference, our study introduces a regularization term that specifically targets the dynamics and population of the mean and variance of the parameters. This term aims to retain the benefits of KL divergence while addressing related challenges. To ensure proper correspondence between network parameters and the data, our method introduces an importance-weighted Evidence Lower Bound term to capture data and parameter correlations. This enables storage of common and distinctive parameter hyperspace bases. The proposed method partitions the parameter space into common and distinctive subspaces, with conditions for effective backward and forward knowledge transfer, elucidating the network-parameter dataset correspondence. The experimental results demonstrate the effectiveness of our method across diverse datasets and various combinations of sequential datasets, yielding superior performance compared to existing approaches.

Multiple-Valued Feedback and Recurrent Correlation Neural Networks.

Multiperiodicity of Periodically Oscillated Discrete-Time Neural Networks with Transient Excitatory Self-Connections and Sigmoidal Nonlinearities

Learning, Storing, and Disentangling Correlated Patterns in Neural Networks.

Hypercomplex-Valued Recurrent Correlation Neural Networks

A SYMMETRIC TERNARY LOGIC BASED MULTI-VALUED LEARNING NETWORK

Dynamics Analysis and Analog Associative Memory of Networks with LT Neurons.

Coexistence and Local Stability of Multiple Equilibria in Neural Networks with Piecewise Linear Nondecreasing Activation Functions

High-Fidelity Coding with Correlated Neurons

Neural information processing with feedback modulations.

Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks

Multiple-Weight Recurrent Neural Networks.

Multiobjective Neural Network Ensembles Based on Regularized Negative Correlation Learning.

Calibrated Multivariate Regression Networks

Deep Multi-View Learning using Neuron-Wise Correlation-Maximizing Regularizers

Revised Regularization for Efficient Continual Learning through Correlation-Based Parameter Update in Bayesian Neural Networks

Enhancing Neural Training via a Correlated Dynamics Model

The UPPAM Continuous-Time RNN Model and Its Critical Dynamics Study

Convergence and periodicity of solutions for a class of discrete-time recurrent neural network with two neurons

Model Predictive Control of Linear Parameter Varying Systems Based on a Recurrent Neural Network.

Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits

Continual Learning by Modeling Intra-Class Variation