Abstract:We consider the problem of transferring knowledge from a source, or proxy, domain to a new target domain for learning a high-dimensional regression model with possibly different features. Recently, the statistical properties of homogeneous transfer learning have been investigated. However, most homogeneous transfer and multi-task learning methods assume that the target and proxy domains have the same feature space, limiting their practical applicability. In applications, target and proxy feature spaces are frequently inherently different, for example, due to the inability to measure some variables in the target data-poor environments. Conversely, existing heterogeneous transfer learning methods do not provide statistical error guarantees, limiting their utility for scientific discovery. We propose a two-stage method that involves learning the relationship between the missing and observed features through a projection step in the proxy data and then solving a joint penalized regression optimization problem in the target data. We develop an upper bound on the method's parameter estimation risk and prediction risk, assuming that the proxy and the target domain parameters are sparsely different. Our results elucidate how estimation and prediction error depend on the complexity of the model, sample size, the extent of overlap, and correlation between matched and mismatched features.

What problem does this paper attempt to address?

This paper attempts to solve the problem of knowledge transfer in high - dimensional regression models when the feature spaces of the target domain and the source domain do not completely match. Specifically, the research focuses on: 1. **Knowledge Transfer under Feature Space Mismatch**: Most existing transfer learning methods assume that the source domain and the target domain have the same feature space, which is often not true in practical applications. For example, in a new data - scarce environment, researchers may not be able to measure certain variables. Therefore, this paper aims to develop a method that can perform knowledge transfer in the case of feature space mismatch. 2. **Statistical Error Guarantees**: Although some heterogeneous transfer learning (HTL) methods have been proposed, these methods usually lack strict statistical error guarantees, which limits their practicality in scientific research. To this end, the author proposes a two - stage method and provides upper bounds for parameter estimation risk and prediction risk. 3. **Improved Estimation and Prediction in High - Dimensional Linear Models**: By combining information from the source domain and the target domain, this method aims to improve the estimation and prediction performance in the target domain. In particular, when the parameter differences between the source domain and the target domain are small, this method can more effectively use the large - scale data in the source domain to make up for the problem of insufficient data in the target domain. ### Specific Problem Description The paper considers how to transfer knowledge from a source domain (or proxy domain) to a new target domain to learn a high - dimensional regression model, while the features of these two domains may be different. Specific challenges include: - **Feature Space Mismatch**: The feature spaces of the target domain and the source domain are different, and some features are not available in the target domain. - **Limited Sample Size**: The amount of data in the target domain is limited, while there is a large amount of available data in the source domain. - **Parameter Differences**: There may be parameter differences between the source domain and the target domain, but these differences are sparse. ### Solution Overview To solve the above problems, the author proposes a two - stage method: 1. **First Stage (Imputation)**: Learn the relationship between missing features and observed features in the source domain through a projection step, and use this relationship in the target domain to estimate the missing features. 2. **Second Stage (Estimation)**: Solve a joint penalized regression optimization problem in the target domain to obtain the final model parameter estimates. In addition, the author also derives the upper bounds of the parameter estimation risk and prediction risk of this method, and analyzes how these errors depend on factors such as model complexity, sample size, and feature matching degree. ### Main Contributions - Proposes a heterogeneous transfer learning method suitable for the case of feature space mismatch. - Provides strict upper bounds of statistical errors, ensuring the theoretical reliability of the method. - Verifies the effectiveness of the method in the case of limited samples through experiments, showing its potential in practical applications. In summary, this paper solves the key problem of efficient knowledge transfer in the case of feature space mismatch and provides a new solution for the estimation and prediction of high - dimensional regression models.

Heterogeneous transfer learning for high dimensional regression with feature mismatch

Transfer Learning for High Dimensional Robust Regression

Heterogeneous Transfer Learning for Building High-Dimensional Generalized Linear Models with Disparate Datasets

A Survey on Heterogeneous Transfer Learning

Estimation and inference for transfer learning with high-dimensional quantile regression

Learning Transferable Convolutional Proxy by SMI-Based Matching Technique

Distributed Heterogeneous Transfer Learning

Improvement of Heterogeneous Transfer Learning Efficiency by Using Hebbian Learning Principle

Transfer learning with high dimensional composite quantile regression

A Recent Survey of Heterogeneous Transfer Learning

Heterogeneous Transfer Learning for Image Classification.

Heterogeneous Spectral-Spatial Feature Transfer With Structure Preserved Distribution Alignment for Hyperspectral Image Classification

Transfer Learning on Heterogeneous Feature Spaces for Treatment Effects Estimation

Robust multiple subspaces transfer for heterogeneous domain adaptation

Homogeneous Online Transfer Learning with Online Distribution Discrepancy Minimization

TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression

Heterogeneous Domain Adaptation Through Progressive Alignment.

Transfer Learning for High-dimensional Quantile Regression via Convolution Smoothing

Transfer Learning for High-dimensional Quantile Regression with Distribution Shift

Transfer Kernel Learning for Multi-Source Transfer Gaussian Process Regression

A Framework for Supervised Heterogeneous Transfer Learning Using Dynamic Distribution Adaptation and Manifold Regularization