Knowledge probing is crucial for understanding the knowledge transfer
mechanism behind the pre-trained language models (PLMs). Despite the growing
progress of probing knowledge for PLMs in the general domain, specialised areas
such as biomedical domain are vastly under-explored. To catalyse the research
in this direction, we release a well-curated biomedical knowledge probing
benchmark, MedLAMA, which is constructed based on the Unified Medical Language
System (UMLS) Metathesaurus. We test a wide spectrum of state-of-the-art PLMs
and probing approaches on our benchmark, reaching at most 3% of acc@10. While
highlighting various sources of domain-specific challenges that amount to this
underwhelming performance, we illustrate that the underlying PLMs have a higher
potential for probing tasks. To achieve this, we propose Contrastive-Probe, a
novel self-supervised contrastive probing approach, that adjusts the underlying
PLMs without using any probing data. While Contrastive-Probe pushes the acc@10
to 28%, the performance gap still remains notable. Our human expert evaluation
suggests that the probing performance of our Contrastive-Probe is still
under-estimated as UMLS still does not include the full spectrum of factual
knowledge. We hope MedLAMA and Contrastive-Probe facilitate further
developments of more suited probing techniques for this domain.

本研究采用对比探针法，探究生物医学领域基于 UMLS 词汇库的预训练语言模型的知识转移机制，并提出了 MedLAMA 作为基准来测试多种最先进的语言模型和探测方法，其中 Contrastive-Probe 方法的性能表现优于其他方法，为此领域更合适的探针技术的发展提供了启示。