Language Models appear to perform poorly on quantification. We ask how badly.
'Few'-type quantifiers, as in 'few children like vegetables' might pose a
particular challenge for Language Models, since the sentence components without
the quantifier are likely to co-occur, and because 'few'-type quantifiers are
rare. We present 960 sentences stimuli from two human neurolinguistic
experiments to 22 autoregressive transformer models of differing sizes. Not
only do the models perform poorly on 'few'-type quantifiers, but overall the
larger the model, the worse its performance. We interpret this inverse scaling
as suggesting that larger models increasingly reflect online rather than
offline human processing, and argue that decreasing performance of larger
models may challenge uses of Language Models as the basis for Natural Language
Systems.

语言模型在量化任务中表现较差，特别是在 Few-type 的量化词上，而且模型越大，表现越差。这暗示着越大的模型越倾向于反映在线而非离线的人类语言处理，并且这种表现下降可能挑战语言模型作为自然语言系统基础的应用。