Large language models (LLMs) show amazing proficiency and fluency in the use
of language. Does this mean that they have also acquired insightful linguistic
knowledge about the language, to an extent that they can serve as an "expert
linguistic annotator"? In this paper, we examine the successes and limitations
of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning
structure, focusing on the Abstract Meaning Representation (AMR; Banarescu et
al. 2013) parsing formalism, which provides rich graphical representations of
sentence meaning structure while abstracting away from surface forms. We
compare models' analysis of this semantic structure across two settings: 1)
direct production of AMR parses based on zero- and few-shot prompts, and 2)
indirect partial reconstruction of AMR via metalinguistic natural language
queries (e.g., "Identify the primary event of this sentence, and the predicate
corresponding to that event."). Across these settings, we find that models can
reliably reproduce the basic format of AMR, and can often capture core event,
argument, and modifier structure -- however, model outputs are prone to
frequent and major errors, and holistic analysis of parse acceptability shows
that even with few-shot demonstrations, models have virtually 0% success in
producing fully accurate parses. Eliciting natural language responses produces
similar patterns of errors. Overall, our findings indicate that these models
out-of-the-box can capture aspects of semantic structure, but there remain key
limitations in their ability to support fully accurate semantic analyses or
parses.

通过比较 GPT-3、ChatGPT 和 GPT-4 模型的分析结果，研究发现这些模型能够可靠地重现 AMR 的基本格式，并且通常能够捕捉到核心事件、论证和修饰结构，但模型的输出容易出现频繁和重大错误，从整体上来看，即使在演示中，模型也几乎没有成功地产生完全准确的解析结果，这表明这些模型尽管能够捕捉到语义结构的某些方面，但在支持完全准确的语义分析或解析方面仍存在关键的局限性。