Pre-trained language models (PLMs) were considered to be able to store
relational knowledge present in the training data. However, some relational
knowledge seems to be discarded unsafely in PLMs due to \textbf{report bias}:
low-frequency relational knowledge might be underexpressed compared to
high-frequency one in PLMs. This gives us a hint that relational knowledge
might not be redundant to the stored knowledge of PLMs, but rather be
complementary. To additionally inject relational knowledge into PLMs, we
propose a simple-yet-effective approach to inject relational knowledge into
PLMs, which is inspired by three observations (namely, polymorphism, synonymous
substitution, and association). In particular, we switch entities in the
training corpus to related entities (either hypernyms/hyponyms/synonyms, or
arbitrarily-related concepts). Experimental results show that the proposed
approach could not only better capture relational knowledge, but also improve
the performance in various biomedical downstream tasks. Our model is available
in https://github.com/StevenZHB/BioPLM_InjectingKnowledge.

通过实验表明，将相关实体替换为同义词、上下位词或相关概念是将关系知识注入预训练语言模型的一种简单有效方法，该方法能够更好地捕捉关系知识，并提高各种生物医学领域下游任务的性能。

通过多态性和同义替换向生物医学预训练模型注入知识

Injecting Knowledge into Biomedical Pre-trained Models via Polymorphism  and Synonymous Substitution

Pre-trained language models (PLMs) have been the de facto paradigm for most
natural language processing (NLP) tasks. This also benefits biomedical domain:
researchers from informatics, medicine, and computer science (CS) communities
propose various PLMs trained on biomedical datasets, e.g., biomedical text,
electronic health records, protein, and DNA sequences for various biomedical
tasks. However, the cross-discipline characteristics of biomedical PLMs hinder
their spreading among communities; some existing works are isolated from each
other without comprehensive comparison and discussions. It expects a survey
that not only systematically reviews recent advances of biomedical PLMs and
their applications but also standardizes terminology and benchmarks. In this
paper, we summarize the recent progress of pre-trained language models in the
biomedical domain and their applications in biomedical downstream tasks.
Particularly, we discuss the motivations and propose a taxonomy of existing
biomedical PLMs. Their applications in biomedical downstream tasks are
exhaustively discussed. At last, we illustrate various limitations and future
trends, which we hope can provide inspiration for the future research of the
research community.

本篇论文综述了近期预训练语言模型在生物医学领域的研究进展及其在生物医学下游任务中的应用，提出了现有生物医学 PLMs 的分类及其在下游任务中的应用，讨论了其限制与未来发展趋势。