Abstract:Continual learning (CL) desires that the neural network sequentially perform learning tasks from a dynamic data stream without forgetting learned knowledge. To overcome forgetting, a line of work relies on gradient projection to minimize the influence between gradients during optimization. This article focuses on a challenging problem concerning CL: When, how, and where to implement gradient projection to promote CL. Tackling this problem can be divided into two perspectives, namely the gradient direction ( when and how ) and the area of gradient conflict ( where ). First, we propose a plug-and-play method UniGrad to tackle the inconsistency of conflicting and nonconflicting gradients during optimization in CL. Second, we explore the interaction mechanism of gradient projection and loss landscape in CL, and further propose a pluggable method UniGrad-FS to improve the CL performance. In short, this work expects to overcome forgetting through an efficient gradient projection at the area where the gradient conflicts are less intense. In essence, the proposed method is a general and pluggable method that can be used in any gradient-based optimizer. For evaluation, we plug UniGrad and UniGrad-FS into two top-performing baselines (WA and MEMO). Our method shows clear improvements, i.e., boosting WA and MEMO by +2.09% and 1.72% in the 20-step of the CIFAR100 benchmark. In addition, we observe performance enhancement on all settings of CIFAR100 and Tiny-ImageNet datasets. Extensive experiments demonstrate the simplicity and effectiveness of the proposed method.

A Fast Supervised Learning Algorithm of Multilayer Feedforward Network Based on Improved Conjugate Gradient Method

UniGrad-FS: Unified Gradient Projection with Flatter Sharpness for Continual Learning

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

A Fast Learning Algorithm for Feedfoward Neural Network Based on the Layer-by-Layer and Neuron-by-Neuron Optimizing Procedure

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks

Fast Learning Algorithms for Feedforward Neural Networks

A Novel Conjugate Gradient Method with Generalized Armijo Search for Efficient Training of Feedforward Neural Networks

An Efficient Conjugate Gradient Based Learning Algorithm for Multiple Optimal Learning Factors of Multilayer Perceptron Neural Network

New Dynamical Optimal Learning for Linear Multilayer FNN

A Fast Learning Strategy For Multilayer Feedforward Neural Networks

A New Conjugate Gradient Method with Smoothing L_1/2 Regularization Based on a Modified Secant Equation for Training Neural Networks

A New Layer-Wise Linearized Algorithm for Feedforward Neural Network

Optimal Algorithm for a Multilayer Feedforward Net and Its Application

An Efficient Generalized Single Hidden Layer Neural Networks Based on Conjugate Gradient Method

An Algorithm for Fast Convergence of Back Propagation by Enlarging Error

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

Convergence of a modified gradient-based learning algorithm with penalty for single-hidden-layer feed-forward networks

A New Conjugate Gradient Method with Smoothing $$L_{1/2} $$L1/2 Regularization Based on a Modified Secant Equation for Training Neural Networks

Fast Computational Approach to the Levenberg-Marquardt Algorithm for Training Feedforward Neural Networks

An adaptive natural gradient method with adaptive step size in multilayer perceptrons

A Fast Learning Algorithm of Feedforward Neural Networks by Using Novel Error Functions