The advent of large-scale pre-trained language models has contributed greatly
to the recent progress in natural language processing. Many state-of-the-art
language models are first trained on a large text corpus and then fine-tuned on
downstream tasks. Despite its recent success and wide adoption, fine-tuning a
pre-trained language model often suffers from overfitting, which leads to poor
generalizability due to the extremely high complexity of the model and the
limited training samples from downstream tasks. To address this problem, we
propose a novel and effective fine-tuning framework, named Layerwise Noise
Stability Regularization (LNSR). Specifically, we propose to inject the
standard Gaussian noise or In-manifold noise and regularize hidden
representations of the fine-tuned model. We first provide theoretical analyses
to support the efficacy of our method. We then demonstrate the advantages of
the proposed method over other state-of-the-art algorithms including L2-SP,
Mixout and SMART. While these previous works only verify the effectiveness of
their methods on relatively simple text classification tasks, we also verify
the effectiveness of our method on question answering tasks, where the target
problem is much more difficult and more training examples are available.
Furthermore, extensive experimental results indicate that the proposed
algorithm can not only enhance the in-domain performance of the language models
but also improve the domain generalization performance on out-of-domain data.

本文提出了一种名为 LNSR 的 fine-tuning 框架，通过注入高斯噪声或浸入式噪声，对 fine-tuned 模型的隐藏表示进行规范化，以解决预训练语言模型的过拟合问题，并证明其在 question answering task 方面具有优越性。