The main challenge that sets transfer learning apart from traditional supervised learning is the distribution shift, reflected as the shift between the source and target models and that between the marginal covariate distributions. In this work, we tackle model shifts in the presence of covariate shifts in the high-dimensional regression setting. Specifically, we propose a two-step method with a novel fused-regularizer that effectively leverages samples from source tasks to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model, showing the robustness of the proposed method to covariate shifts. We further establish conditions under which the estimator is minimax-optimal. Additionally, we extend the method to a distributed setting, allowing for a pretraining-finetuning strategy, requiring just one round of communication while retaining the estimation rate of the centralized version. Numerical tests validate our theory, highlighting the method's robustness to covariate shifts.

在高维回归环境中，我们提出了一种具有新型融合正则化器的两步法，有效利用来自源任务的样本，提高对具有有限样本的目标任务的学习性能，并提供了目标模型估计误差的非渐近界限，表明所提方法对协变量转变的鲁棒性。我们进一步确定了估计器最小化优选的条件。此外，我们将该方法扩展到分布式环境，允许预训练和微调策略，仅需一轮通信，同时保持了集中式版本的估计率。数值测试验证了我们的理论，突出了该方法对协变量转变的鲁棒性。

TransFusion：高维回归的协变量漂移鲁棒迁移学习