Abstract:Transfer learning algorithms have been developed in various applicational contexts while only a few of them offer statistical guarantees in high-dimensions. Among these work, the differences between the target and sources, a.k.a. the contrasts, are typically modeled as, or at least close to, vectors with certain low-dimensional structure (e.g., sparsity), resulting in a separate debiasing step after a preceding pooling estimation procedure. Under such intuitive yet powerful framework, additional homogeneity conditions on Hessian matrices of the population loss functions are often imposed to preserve the delicate low-dimensional structure of the contrasts during pooling, which is either unrealistic in practice or easily destroyed by basic data transformation such as standardization. In this article, under the general M-estimators framework with decomposable regularizers, we highlight the role of fine-tuning underneath the conspicuous gain of the debiasing step in transfer learning. Namely, we find it is possible to enhance estimation accuracy by fine-tuning a primal estimator sufficiently close to the true target one. Our theory suggests slightly enlarging the pooling regularization strength when either the contrast's low-dimensional structure or the homogeneity of Hessian matrices is violated. Traditional linear regression and generalized low-rank trace regression in high-dimensions are discussed as two specific examples under our framework. When the informative source datasets are unknown, a novel truncated-penalized algorithm is proposed to directly output the primal estimator by simultaneously selecting the useful sources and its oracle property is proved. Extensive numerical experiments are conducted to validate the theoretical assertions. A case study on the air quality regulation in China by transfer learning is also provided for illustration.

Transfer learning for piecewise-constant mean estimation: Optimality, $\ell_1$- and $\ell_0$-penalisation

Transfer Learning for Functional Mean Estimation: Phase Transition and Adaptive Algorithms

The Role of Fine-tuning: Transfer Learning for High-dimensional M-estimators with Decomposable Regularizers

The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression

Transfer Learning with General Estimating Equations

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning

Generalization error of min-norm interpolators in transfer learning

Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier

Transfer Learning in $\ell_1$ Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis

Simultaneous Estimation and Dataset Selection for Transfer Learning in High Dimensions by a Non-convex Penalty

Transfer Learning for Nonparametric Regression: Non-asymptotic Minimax Analysis and Adaptive Procedure

A Statistical Theory of Regularization-Based Continual Learning

Estimation and inference for transfer learning with high-dimensional quantile regression

$L_0$-Estimation of Piecewise-Constant Signals on Graphs

Weighted likelihood transfer learning for high-dimensional generalized linear models

Transfer Learning for High-dimensional Quantile Regression via Convolution Smoothing

Understanding Transfer Learning via Mean-field Analysis

Transfer Learning for High-Dimensional Quantile Regression Via Convolution Smoothing

AdaTrans: Feature-wise and Sample-wise Adaptive Transfer Learning for High-dimensional Regression

Transferability Estimation Based on Principal Gradient Expectation.