Pre-trained language models (LMs) have become ubiquitous in solving various
natural language processing (NLP) tasks. There has been increasing interest in
what knowledge these LMs contain and how we can extract that knowledge,
treating LMs as knowledge bases (KBs). While there has been much work on
probing LMs in the general domain, there has been little attention to whether
these powerful LMs can be used as domain-specific KBs. To this end, we create
the BioLAMA benchmark, which is comprised of 49K biomedical factual knowledge
triples for probing biomedical LMs. We find that biomedical LMs with recently
proposed probing methods can achieve up to 18.51% Acc@5 on retrieving
biomedical knowledge. Although this seems promising given the task difficulty,
our detailed analyses reveal that most predictions are highly correlated with
prompt templates without any subjects, hence producing similar results on each
relation and hindering their capabilities to be used as domain-specific KBs. We
hope that BioLAMA can serve as a challenging benchmark for biomedical factual
probing.

通过创建生物医学知识三元组的基准测试 BioLAMA，研究了预训练语言模型是否可用作具有生物医学特定领域知识的知识库。发现在近期提出的探测方法下，生物医学语言模型可以在检索生物医学知识方面取得 18.51% 的精确度，但多数预测与无主题的提示模板高度相关，因此限制了它们作为特定领域知识库的能力。