When large language models (LMs) are applied in zero- or few-shot settings to
discriminative tasks such as multiple-choice questions, their attentiveness
(i.e., probability mass) is spread across many vocabulary tokens that are not
valid choices. Such a spread across multiple surface forms with identical
meaning is thought to cause an underestimation of a model's true performance,
referred to as the "surface form competition" (SFC) hypothesis. This has
motivated the introduction of various probability normalization methods.
However, many core questions remain unanswered. How do we measure SFC or
attentiveness? Are there direct ways of increasing attentiveness on valid
choices? Does increasing attentiveness always improve task accuracy? We propose
a mathematical formalism for studying this phenomenon, provide a metric for
quantifying attentiveness, and identify a simple method for increasing it --
namely, in-context learning with even just one example containing answer
choices. The formalism allows us to quantify SFC and bound its impact. Our
experiments on three diverse datasets and six LMs reveal several surprising
findings. For example, encouraging models to generate a valid answer choice
can, in fact, be detrimental to task performance for some LMs, and prior
probability normalization methods are less effective (sometimes even
detrimental) to instruction-tuned LMs. We conclude with practical insights for
effectively using prompted LMs for multiple-choice tasks.

研究了大型语言模型的注意力问题对其在多项选择任务中性能的影响，提出了一种简单有效的增加注意力的方法，并得出了一些实用的结论。