It has become standard to solve NLP tasks by fine-tuning pre-trained language
models (LMs), especially in low-data settings. There is minimal theoretical
understanding of empirical success, e.g., why fine-tuning a model with $10^8$
or more parameters on a couple dozen training points does not result in
overfitting. We investigate whether the Neural Tangent Kernel (NTK) - which
originated as a model to study the gradient descent dynamics of infinitely wide
networks with suitable random initialization - describes fine-tuning of
pre-trained LMs. This study was inspired by the decent performance of NTK for
computer vision tasks (Wei et al., 2022). We extend the NTK formalism to Adam
and use Tensor Programs (Yang, 2020) to characterize conditions under which the
NTK lens may describe fine-tuning updates to pre-trained language models.
Extensive experiments on 14 NLP tasks validate our theory and show that
formulating the downstream task as a masked word prediction problem through
prompting often induces kernel-based dynamics during fine-tuning. Finally, we
use this kernel view to propose an explanation for the success of
parameter-efficient subspace-based fine-tuning methods.

通过对 14 个 NLP 任务的实验验证，我们探究了采用 NTK 对预训练语言模型进行 fine-tuning 的机制和条件，并提出了一个基于 NTK 的解释，解释了参数高效子空间 fine-tuning 方法的成功原因。