With their increasing size, Large language models (LLMs) are becoming
increasingly good at language understanding tasks. But even with high
performance on specific downstream task, LLMs fail at simple linguistic tests
for negation or quantifier understanding. Previous work on testing capability
of LLMs on understanding quantifiers suggest that as the size of the models
increase, they get better at understanding most-type quantifiers but get
increasingly worse at understanding few-type quantifiers, thus presenting a
case of an inverse-scaling law. In this paper, we question the claims of
inverse scaling of few-type quantifier understanding in LLMs and show that it
is a result of inappropriate testing methodology. We also present alternate
methods to measure quantifier comprehension in LLMs and show that as the size
of the models increase, these behaviours are different from what is shown in
previous research. LLMs are consistently able to understand the difference
between the meaning of few-type and most-type quantifiers, but when a
quantifier is added to phrase, LLMs do not always take into account the meaning
of the quantifier. We in fact see an inverse scaling law for most-type
quantifiers, which is contrary to human psycho-linguistic experiments and
previous work, where the model's understanding of most-type quantifier gets
worse as the model size increases. We do this evaluation on models ranging from
125M-175B parameters, which suggests that LLMs do not do as well as expected
with quantifiers and statistical co-occurrence of words still takes precedence
over word meaning.

本文探讨了大型语言模型在量词理解方面的能力，发现传统的测试方法导致了对 LLM 在 few-type quantifiers 上理解能力逆反缩放的错误认知。研究者提出了替代方法，发现随着模型规模的增加，LLM 在量词理解方面的表现与之前的研究不同，并揭示了 most-type quantifiers 存在逆反缩放的特性，这与人类语言认知实验和以前的研究相反。

大型语言模型中的量词理解探究

Probing Quantifier Comprehension in Large Language Models

Language Models appear to perform poorly on quantification. We ask how badly.
'Few'-type quantifiers, as in 'few children like vegetables' might pose a
particular challenge for Language Models, since the sentence components without
the quantifier are likely to co-occur, and because 'few'-type quantifiers are
rare. We present 960 sentences stimuli from two human neurolinguistic
experiments to 22 autoregressive transformer models of differing sizes. Not
only do the models perform poorly on 'few'-type quantifiers, but overall the
larger the model, the worse its performance. We interpret this inverse scaling
as suggesting that larger models increasingly reflect online rather than
offline human processing, and argue that decreasing performance of larger
models may challenge uses of Language Models as the basis for Natural Language
Systems.

语言模型在量化任务中表现较差，特别是在 Few-type 的量化词上，而且模型越大，表现越差。这暗示着越大的模型越倾向于反映在线而非离线的人类语言处理，并且这种表现下降可能挑战语言模型作为自然语言系统基础的应用。