Advances in computational methods and big data availability have recently translated into breakthroughs in AI applications. With successes in bottom-up challenges partially overshadowing shortcomings, the 'human-like' performance of Large Language Models has raised the question of how linguistic performance is achieved by algorithms. Given systematic shortcomings in generalization across many AI systems, in this work we ask whether linguistic performance is indeed guided by language knowledge in Large Language Models. To this end, we prompt GPT-3 with a grammaticality judgement task and comprehension questions on less frequent constructions that are thus unlikely to form part of Large Language Models' training data. These included grammatical 'illusions', semantic anomalies, complex nested hierarchies and self-embeddings. GPT-3 failed for every prompt but one, often offering answers that show a critical lack of understanding even of high-frequency words used in these less frequent grammatical constructions. The present work sheds light on the boundaries of the alleged AI human-like linguistic competence and argues that, far from human-like, the next-word prediction abilities of LLMs may face issues of robustness, when pushed beyond training data.

本研究探讨基于计算方法和大数据的人工智能应用中所使用的大型语言模型的语言表现，着重考察语言表现是否由语言知识所导引。结果发现即便是常规语法结构之外的、不太常见的语言结构也可能导致大型语言模型表现出不足的语言能力，暗示这种类型的语言模型实际上并不像人类一样具有对语言的深入理解。

在测试语言不常见方面的人工智能性能时，暴露对底层意义的不敏感性