Large Language Models (LLMs) play a crucial role in capturing structured
semantics to enhance language understanding, improve interpretability, and
reduce bias. Nevertheless, an ongoing controversy exists over the extent to
which LLMs can grasp structured semantics. To assess this, we propose using
Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to
extract structured semantics. In our assessment, we employ the prompting
approach, which leads to the creation of our few-shot SRL parser, called
PromptSRL. PromptSRL enables LLMs to map natural languages to explicit semantic
structures, which provides an interpretable window into the properties of LLMs.
We find interesting potential: LLMs can indeed capture semantic structures, and
scaling-up doesn't always mirror potential. Additionally, limitations of LLMs
are observed in C-arguments, etc. Lastly, we are surprised to discover that
significant overlap in the errors is made by both LLMs and untrained humans,
accounting for almost 30% of all errors.

利用语义角色标注 (SRL) 作为基本任务评估了大型语言模型 (LLM) 理解和提取结构化语义的能力。我们提出了 PromptSRL，这是一个少样本 SRL 解析器，可以让 LLM 将自然语言映射到显式的语义结构，从而揭示 LLM 的属性。结果发现，LLM 确实能够捕捉语义结构，扩大模型规模并不总是表现出更好的潜力。此外，我们观察到 LLM 在 C-arguments 等方面存在局限性。最后，令人惊讶的是，在 30% 的错误中，LLM 和未经训练的人类犯了很多重叠的错误。