Recent advances in pre-trained language modeling have facilitated significant
progress across various natural language processing (NLP) tasks. Word masking
during model training constitutes a pivotal component of language modeling in
architectures like BERT. However, the prevalent method of word masking relies
on random selection, potentially disregarding domain-specific linguistic
attributes. In this article, we introduce an innovative masking approach
leveraging genre and topicality information to tailor language models to
specialized domains. Our method incorporates a ranking process that prioritizes
words based on their significance, subsequently guiding the masking procedure.
Experiments conducted using continual pre-training within the legal domain have
underscored the efficacy of our approach on the LegalGLUE benchmark in the
English language. Pre-trained language models and code are freely available for
use.

在本文中，我们介绍了一种创新的掩码方法，利用流派和主题信息来定制语言模型以适用于专业领域，该方法基于词语的重要性对其进行排名，并优先选择重要性高的词进行掩码处理，通过在法律领域进行连续预训练实验证明了我们的方法在 LegalGLUE 基准测试中的有效性，同时提供免费使用的预训练语言模型和代码。