We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-style pretraining, the main encoder is trained as a discriminator to detect replaced tokens generated by auxiliary masked language models (MLMs). Different from ELECTRA which trains one MLM as the generator, we jointly train multiple MLMs of different sizes to provide training signals at various levels of difficulty. To push the discriminator to learn better with challenging replaced tokens, we learn mixture weights over the auxiliary MLMs' outputs to maximize the discriminator loss by backpropagating the gradient from the discriminator via Gumbel-Softmax. For better pretraining efficiency, we propose a way to assemble multiple MLMs into one unified auxiliary model. AMOS outperforms ELECTRA and recent state-of-the-art pretrained models by about 1 point on the GLUE benchmark for BERT base-sized models.

提出了一种新的框架AMOS，其中通过多个辅助生成器的多个信号混合来用对抗学习课程进行文本编码器的预训练，通过ELECTRA风格的预训练，主编码器训练为鉴别器，以检测辅助掩蔽语言模型生成的替换标记，AMOS在GLUE基准测试中比ELECTRA和最新的最先进的预训练模型表现优秀。

利用对抗式训练信号生成器进行文本编码器的预训练