Pre-trained neural language models bring significant improvement for various
NLP tasks, by fine-tuning the models on task-specific training sets. During
fine-tuning, the parameters are initialized from pre-trained models directly,
which ignores how the learning process of similar NLP tasks in different
domains is correlated and mutually reinforced. In this paper, we propose an
effective learning procedure named Meta Fine-Tuning (MFT), served as a
meta-learner to solve a group of similar NLP tasks for neural language models.
Instead of simply multi-task training over all the datasets, MFT only learns
from typical instances of various domains to acquire highly transferable
knowledge. It further encourages the language model to encode domain-invariant
representations by optimizing a series of novel domain corruption loss
functions. After MFT, the model can be fine-tuned for each domain with better
parameter initializations and higher generalization ability. We implement MFT
upon BERT to solve several multi-domain text mining tasks. Experimental results
confirm the effectiveness of MFT and its usefulness for few-shot learning.

本文提出了一种名为 Meta Fine-Tuning (MFT) 的有效学习过程，该过程可作为元学习器解决多个相似的自然语言处理任务，以获取高度可转移的知识，并通过优化一系列新颖的领域破坏损失函数来鼓励语言模型对领域不变表示进行编码。实验证明，MFT 对于小样本学习具有很大的用途。