Estimating the log-likelihood of a given sentence under an autoregressive
language model is straightforward: one can simply apply the chain rule and sum
the log-likelihood values for each successive token. However, for masked
language models, there is no direct way to estimate the log-likelihood of a
sentence. To address this issue, Salazar et al. (2020) propose to estimate
sentence pseudo-log-likelihood (PLL) scores, computed by successively masking
each sentence token, retrieving its score using the rest of the sentence as
context, and summing the resulting values. Here, we demonstrate that the
original PLL method yields inflated scores for out-of-vocabulary words and
propose an adapted metric, in which we mask not only the target token, but also
all within-word tokens to the right of the target. We show that our adapted
metric (PLL-word-l2r) outperforms both the original PLL metric and a PLL metric
in which all within-word tokens are masked. In particular, it better satisfies
theoretical desiderata and better correlates with scores from autoregressive
models. Finally, we show that the choice of metric affects even tightly
controlled, minimal pair evaluation benchmarks (such as BLiMP), underscoring
the importance of selecting an appropriate scoring metric for evaluating MLM
properties.

该论文提出了一种适用于遮蔽语言模型的评估方法，该方法通过遮蔽目标词汇和目标词汇右侧的所有词汇来计算伪对数似然分数，并表明该方法优于原始的伪对数似然方法和一种遮蔽单词内词汇的伪对数似然方法，并且选择适当的评分指标对于评估一个语言模型的性能很重要。