Large Language Models (LLMs) play a crucial role in capturing structured
semantics to enhance language understanding, improve interpretability, and
reduce bias. Nevertheless, an ongoing controversy exists over the extent to
which LLMs can grasp structured semantics. To assess this, we propose using
Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to
extract structured semantics. In our assessment, we employ the prompting
approach, which leads to the creation of our few-shot SRL parser, called
PromptSRL. PromptSRL enables LLMs to map natural languages to explicit semantic
structures, which provides an interpretable window into the properties of LLMs.
We find interesting potential: LLMs can indeed capture semantic structures, and
scaling-up doesn't always mirror potential. Additionally, limitations of LLMs
are observed in C-arguments, etc. Lastly, we are surprised to discover that
significant overlap in the errors is made by both LLMs and untrained humans,
accounting for almost 30% of all errors.

利用语义角色标注 (SRL) 作为基本任务评估了大型语言模型 (LLM) 理解和提取结构化语义的能力。我们提出了 PromptSRL，这是一个少样本 SRL 解析器，可以让 LLM 将自然语言映射到显式的语义结构，从而揭示 LLM 的属性。结果发现，LLM 确实能够捕捉语义结构，扩大模型规模并不总是表现出更好的潜力。此外，我们观察到 LLM 在 C-arguments 等方面存在局限性。最后，令人惊讶的是，在 30% 的错误中，LLM 和未经训练的人类犯了很多重叠的错误。

LLM 对于捕捉结构化语义的潜力与限制：以 SRL 为例的案例研究

Potential and Limitations of LLMs in Capturing Structured Semantics: A  Case Study on SRL

Recent advancement in the capabilities of large language models (LLMs) has
triggered a new surge in LLMs' evaluation. Most recent evaluation works tends
to evaluate the comprehensive ability of LLMs over series of tasks. However,
the deep structure understanding of natural language is rarely explored. In
this work, we examine the ability of LLMs to deal with structured semantics on
the tasks of question answering with the help of the human-constructed formal
language. Specifically, we implement the inter-conversion of natural and formal
language through in-context learning of LLMs to verify their ability to
understand and generate the structured logical forms. Extensive experiments
with models of different sizes and in different formal languages show that
today's state-of-the-art LLMs' understanding of the logical forms can approach
human level overall, but there still are plenty of room in generating correct
logical forms, which suggest that it is more effective to use LLMs to generate
more natural language training data to reinforce a small model than directly
answering questions with LLMs. Moreover, our results also indicate that models
exhibit considerable sensitivity to different formal languages. In general, the
formal language with the lower the formalization level, i.e. the more similar
it is to natural language, is more LLMs-friendly.

最近大规模语言模型能力的进步引发了对其评估的新浪潮，这篇研究工作通过在自然语言和形式语言之间的相互转换来验证大规模语言模型理解和生成结构化逻辑形式的能力，实验证明现今最先进的大规模语言模型在理解逻辑形式方面整体上接近人类水平，但在生成正确逻辑形式方面仍有改进的空间，使用大规模语言模型生成更自然的语言训练数据以增强小型模型的效果更好，同时结果还表明模型对不同形式语言表现出显著的敏感性，总体而言，形式化程度较低、更接近自然语言的形式语言对大规模语言模型更友好。