We propose a method to automatically generate a domain- and task-adaptive
maskings of the given text for self-supervised pre-training, such that we can
effectively adapt the language model to a particular target task (e.g. question
answering). Specifically, we present a novel reinforcement learning-based
framework which learns the masking policy, such that using the generated masks
for further pre-training of the target language model helps improve task
performance on unseen texts. We use off-policy actor-critic with entropy
regularization and experience replay for reinforcement learning, and propose a
Transformer-based policy network that can consider the relative importance of
words in a given text. We validate our Neural Mask Generator (NMG) on several
question answering and text classification datasets using BERT and DistilBERT
as the language models, on which it outperforms rule-based masking strategies,
by automatically learning optimal adaptive maskings.

本文提出一种基于强化学习的框架，使用自适应的掩码技术用于自我监督预训练，以有效地将语言模型适应到指定目标任务，例如问答。我们在几个问答和文本分类数据集上使用 BERT 和 DistilBERT 作为语言模型，使用离策略演员 - 评论家，熵正则化和经验回放进行强化学习，并使用能考虑文本中词汇相对重要性的基于 Transformer 的策略网络进行学习。使用所提出的神经掩码生成器（NMG）自动生成优化的自适应掩码，比基于规则的掩码策略更优秀。