Domain-specific contextualized language models have demonstrated substantial
effectiveness gains for domain-specific downstream tasks, like similarity
matching, entity recognition or information retrieval. However successfully
applying such models in highly specific language domains requires domain
adaptation of the pre-trained models. In this paper we propose the empirically
motivated Linguistically Informed Masking (LIM) method to focus
domain-adaptative pre-training on the linguistic patterns of patents, which use
a highly technical sublanguage. We quantify the relevant differences between
patent, scientific and general-purpose language and demonstrate for two
different language models (BERT and SciBERT) that domain adaptation with LIM
leads to systematically improved representations by evaluating the performance
of the domain-adapted representations of patent language on two independent
downstream tasks, the IPC classification and similarity matching. We
demonstrate the impact of balancing the learning from different information
sources during domain adaptation for the patent domain. We make the source code
as well as the domain-adaptive pre-trained patent language models publicly
available at this https URL

该研究提出了一种基于语言学启发的遮蔽方法，称为 LIM，该方法可用于域适应的预训练模型，以便在专利语言领域提供更好的表示。通过在两个不同的语言模型上评估专利语言的领域适应表示的性能，以专利 IPC 分类和相似性匹配作为后续任务，该研究证明了该方法的改进效果，并公开了源代码和域适应预训练专利语言模型。