ELECTRA, the generator-discriminator pre-training framework, has achieved
impressive semantic construction capability among various downstream tasks.
Despite the convincing performance, ELECTRA still faces the challenges of
monotonous training and deficient interaction. Generator with only masked
language modeling (MLM) leads to biased learning and label imbalance for
discriminator, decreasing learning efficiency; no explicit feedback loop from
discriminator to generator results in the chasm between these two components,
underutilizing the course learning. In this study, a multi-perspective course
learning (MCL) method is proposed to fetch a many degrees and visual angles for
sample-efficient pre-training, and to fully leverage the relationship between
generator and discriminator. Concretely, three self-supervision courses are
designed to alleviate inherent flaws of MLM and balance the label in a
multi-perspective way. Besides, two self-correction courses are proposed to
bridge the chasm between the two encoders by creating a "correction notebook"
for secondary-supervision. Moreover, a course soups trial is conducted to solve
the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained
model. Experimental results show that our method significantly improves
ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on
GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style
models under the same settings. The pre-trained MCL model is available at
this https URL

本文针对 ELECTRA 预训练生成器与判别器训练时出现的单调训练和交互不足的问题，提出了一种多角度课程学习（MCL）算法。该算法通过将 MLM 缺点和标注标签平衡考虑在内的三个自监督课程设计，解决了单调训练和标签不平衡等问题。由于判别器与生成器无明确的反馈循环，该算法提出了两个自我修正课程，以创建 “修正笔记本” 进行辅助监督，弥合两个编码器之间的差距。生动实验表明，该算法使平均表现提高 2.8％和 3.2％，超过了同样设置下最近的先进 ELECTRA 样式模型。