Abstract:We provide a statistical analysis of regularization-based continual learning on a sequence of linear regression tasks, with emphasis on how different regularization terms affect the model performance. We first derive the convergence rate for the oracle estimator obtained as if all data were available simultaneously. Next, we consider a family of generalized $\ell_2$-regularization algorithms indexed by matrix-valued hyperparameters, which includes the minimum norm estimator and continual ridge regression as special cases. As more tasks are introduced, we derive an iterative update formula for the estimation error of generalized $\ell_2$-regularized estimators, from which we determine the hyperparameters resulting in the optimal algorithm. Interestingly, the choice of hyperparameters can effectively balance the trade-off between forward and backward knowledge transfer and adjust for data heterogeneity. Moreover, the estimation error of the optimal algorithm is derived explicitly, which is of the same order as that of the oracle estimator. In contrast, our lower bounds for the minimum norm estimator and continual ridge regression show their suboptimality. A byproduct of our theoretical analysis is the equivalence between early stopping and generalized $\ell_2$-regularization in continual learning, which may be of independent interest. Finally, we conduct experiments to complement our theory.

Error Analysis on Regularized Regression Based on the Maximum Correntropy Criterion

A Regularized Correntropy Framework for Robust Pattern Recognition

Learning with the Maximum Correntropy Criterion Induced Losses for Regression.

Expectile regression for analyzing heteroscedasticity in high dimension

Kernel Matrix Learning with a General Regularized Risk Functional Criterion

Learning Theory Estimates for Coefficient-Based Regularized Regression

Optimal Rates for Coefficient-Based Regularized Regression

Concentration Estimates for Learning with ℓ1-Regularizer and Data Dependent Hypothesis Spaces

Precise Error Analysis of Regularized M-estimators in High-dimensions

Learning rates for regularized least squares ranking algorithm

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Estimation with Norm Regularization

Regularization in kernel learning

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Robust Regularized Kernel Regression.

Optimal Learning Rates for Distribution Regression

Robust Regularized Low-Rank Matrix Models for Regression and Classification

Kernel regression, minimax rates and effective dimensionality: beyond the regular case

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

A Statistical Theory of Regularization-Based Continual Learning