Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations, but incurs substantial training costs. In this paper, we propose a novel concept-based curriculum masking (CCM) method to efficiently pre-train a language model. CCM has two key differences from existing curriculum learning approaches to effectively reflect the nature of MLM. First, we introduce a carefully-designed linguistic difficulty criterion that evaluates the MLM difficulty of each token. Second, we construct a curriculum that gradually masks words related to the previously masked words by retrieving a knowledge graph. Experimental results show that CCM significantly improves pre-training efficiency. Specifically, the model trained with CCM shows comparative performance with the original BERT on the General Language Understanding Evaluation benchmark at half of the training cost.

本文提出了一种基于概念的课程掩模方法 (CCM) 来高效地预训练一种语言模型，该方法通过引入计算语言学困难度的标准以及构建逐渐屏蔽相关词汇的课程而与现有课程学习方法不同，实验结果表明，使用 CCM 进行的模型预训练在培训成本达到一半时就可与原始 BERT 在 GLUE 基准测试中取得类似的性能。

基于概念的课程屏蔽的遮蔽语言模型高效预训练