In the transfer learning paradigm models learn useful representations (or
features) during a data-rich pretraining stage, and then use the pretrained
representation to improve model performance on data-scarce downstream tasks. In
this work, we explore transfer learning with the goal of optimizing downstream
performance. We introduce a simple linear model that takes as input an
arbitrary pretrained feature transform. We derive exact asymptotics of the
downstream risk and its fine-grained bias-variance decomposition. Our finding
suggests that using the ground-truth featurization can result in
"double-divergence" of the asymptotic risk, indicating that it is not
necessarily optimal for downstream performance. We then identify the optimal
pretrained representation by minimizing the asymptotic downstream risk averaged
over an ensemble of downstream tasks. Our analysis reveals the relative
importance of learning the task-relevant features and structures in the data
covariates and characterizes how each contributes to controlling the downstream
risk from a bias-variance perspective. Moreover, we uncover a phase transition
phenomenon where the optimal pretrained representation transitions from hard to
soft selection of relevant features and discuss its connection to principal
component regression.

在转移学习范式下，该研究探索了使用预训练特征转换来优化下游性能的简单线性模型，并发现了通过最小化沿着一组下游任务的平均渐近下游风险来确定最佳预先训练表示的相对重要性以及从偏差 - 方差角度对它们的贡献的特征和结构的学习。