Language models (LMs) have been argued to overlap substantially with human
beings in grammaticality judgment tasks. But when humans systematically make
errors in language processing, should we expect LMs to behave like cognitive
models of language and mimic human behavior? We answer this question by
investigating LMs' more subtle judgments associated with "language illusions"
-- sentences that are vague in meaning, implausible, or ungrammatical but
receive unexpectedly high acceptability judgments by humans. We looked at three
illusions: the comparative illusion (e.g. "More people have been to Russia than
I have"), the depth-charge illusion (e.g. "No head injury is too trivial to be
ignored"), and the negative polarity item (NPI) illusion (e.g. "The hunter who
no villager believed to be trustworthy will ever shoot a bear"). We found that
probabilities represented by LMs were more likely to align with human judgments
of being "tricked" by the NPI illusion which examines a structural dependency,
compared to the comparative and the depth-charge illusions which require
sophisticated semantic understanding. No single LM or metric yielded results
that are entirely consistent with human behavior. Ultimately, we show that LMs
are limited both in their construal as cognitive models of human language
processing and in their capacity to recognize nuanced but critical information
in complicated language materials.

语言模型的能力与人类在语法判断任务中的重叠度已被讨论，但当人类在语言处理中系统性地出现错误时，我们是否应该期望语言模型像语言认知模型一样模仿人类行为？通过研究与 “语言幻觉” 相关的语言模型更微妙的判断，本文回答了这个问题。研究发现，与需要复杂语义理解的比较幻觉和深度冲击幻觉相比，语言模型所代表的概率更可能与人类对结构依赖的 “负极性项目幻觉” 误判一致。没有一个单独的语言模型或度量方法能完全一致地与人类行为相符。最终，本文展示了语言模型在作为人类语言处理的认知模型以及识别复杂语言材料中微妙但关键信息的能力受到限制。