Fine-tuning pre-trained language models such as BERT has become a common
practice dominating leaderboards across various NLP tasks. Despite its recent
success and wide adoption, this process is unstable when there are only a small
number of training samples available. The brittleness of this process is often
reflected by the sensitivity to random seeds. In this paper, we propose to
tackle this problem based on the noise stability property of deep nets, which
is investigated in recent literature (Arora et al., 2018; Sanyal et al., 2020).
Specifically, we introduce a novel and effective regularization method to
improve fine-tuning on NLP tasks, referred to as Layer-wise Noise Stability
Regularization (LNSR). We extend the theories about adding noise to the input
and prove that our method gives a stabler regularization effect. We provide
supportive evidence by experimentally confirming that well-performing models
show a low sensitivity to noise and fine-tuning with LNSR exhibits clearly
higher generalizability and stability. Furthermore, our method also
demonstrates advantages over other state-of-the-art algorithms including L2-SP
(Li et al., 2018), Mixout (Lee et al., 2020) and SMART (Jiang et al., 2020).

本文提出了一种新颖有效的正则化方法，称为分层噪声稳定性正则化 (LNSR)，其可改善在自然语言处理任务中的 fine-tuning。通过实验证明，使用 LNSR 的 fine-tuning 展现出明显的泛化和稳定性优势，并且我们的方法也显示出优于其他现有算法的优势。