Pre-trained language models have achieved state-of-the-art accuracies on
various text classification tasks, e.g., sentiment analysis, natural language
inference, and semantic textual similarity. However, the reliability of the
fine-tuned text classifiers is an often underlooked performance criterion. For
instance, one may desire a model that can detect out-of-distribution (OOD)
samples (drawn far from training distribution) or be robust against domain
shifts. We claim that one central obstacle to the reliability is the
over-reliance of the model on a limited number of keywords, instead of looking
at the whole context. In particular, we find that (a) OOD samples often contain
in-distribution keywords, while (b) cross-domain samples may not always contain
keywords; over-relying on the keywords can be problematic for both cases. In
light of this observation, we propose a simple yet effective fine-tuning
method, coined masked keyword regularization (MASKER), that facilitates
context-based prediction. MASKER regularizes the model to reconstruct the
keywords from the rest of the words and make low-confidence predictions without
enough context. When applied to various pre-trained language models (e.g.,
BERT, RoBERTa, and ALBERT), we demonstrate that MASKER improves OOD detection
and cross-domain generalization without degrading classification accuracy. Code
is available at this https URL

本文提出一种名为 MASKER 的 fine-tuning 方法，该方法通过关键词正则化来实现基于上下文的预测，提高了预训练语言模型在 OOD 检测和跨域泛化方面的性能。