The differences between cloze-task language model (LM) probing with 1)
expert-made templates and 2) naturally-occurring text have often been
overlooked. Here, we evaluate 16 different LMs on 10 probing English datasets
-- 4 template-based and 6 template-free -- in general and biomedical domains to
answer the following research questions: (RQ1) Do model rankings differ between
the two approaches? (RQ2) Do models' absolute scores differ between the two
approaches? (RQ3) Do the answers to RQ1 and RQ2 differ between general and
domain-specific models? Our findings are: 1) Template-free and template-based
approaches often rank models differently, except for the top domain-specific
models. 2) Scores decrease by up to 42% Acc@1 when comparing parallel
template-free and template-based prompts. 3) Perplexity is negatively
correlated with accuracy in the template-free approach, but,
counter-intuitively, they are positively correlated for template-based probing.
4) Models tend to predict the same answers frequently across prompts for
template-based probing, which is less common when employing template-free
techniques.

通过比较专家制定的模板和自然发生的文本对填空任务语言模型进行探测的差异，我们评估了 16 种不同的模型在 10 个针对英语数据集的探测中，其中 4 个基于模板，6 个基于无模板，以回答以下研究问题：（RQ1）两种方法之间的模型排名是否不同？（RQ2）两种方法之间的模型得分是否不同？（RQ3）在一般和领域特定模型之间，RQ1 和 RQ2 的答案是否不同？我们的发现是：1）除了顶级领域特定模型外，无模板和基于模板的方法通常会对模型进行不同的排名。2）在比较并行的无模板和基于模板提示时，准确率下降了最多 42%。3）在无模板方法中，困惑度与准确性呈负相关，但引人注目的是，对于基于模板的探测，它们呈正相关。4）在基于模板的探测中，模型倾向于经常对不同提示进行相同的答案预测，而在使用无模板技术时这种情况较少见。