Pretrained language models (PLMs) have motivated research on what kinds of
knowledge these models learn. Fill-in-the-blanks problem (e.g., cloze tests) is
a natural approach for gauging such knowledge. BioLAMA generates prompts for
biomedical factual knowledge triples and uses the Top-k accuracy metric to
evaluate different PLMs' knowledge. However, existing research has shown that
such prompt-based knowledge probing methods can only probe a lower bound of
knowledge. Many factors like prompt-based probing biases make the LAMA
benchmark unreliable and unstable. This problem is more prominent in BioLAMA.
The severe long-tailed distribution in vocabulary and large-N-M relation make
the performance gap between LAMA and BioLAMA remain notable. To address these,
we introduce context variance into the prompt generation and propose a new
rank-change-based evaluation metric. Different from the previous known-unknown
evaluation criteria, we propose the concept of "Misunderstand" in LAMA for the
first time. Through experiments on 12 PLMs, our context variance prompts and
Understand-Confuse-Misunderstand (UCM) metric makes BioLAMA more friendly to
large-N-M relations and rare relations. We also conducted a set of control
experiments to disentangle "understand" from just "read and copy".

本文提出了一种新的上下文变量提示和理解 - 困惑 - 误解 (UCM) 度量，以改善 PLMs 在生物医学知识三元组上的性能，并提出新颖的 “误解” 概念来评估不同 PLMs 学习的知识，实验表明这可以使 BioLAMA 对大型 N-M 关系和稀有关系更友好。