We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge
set for Abstract Meaning Representation (AMR) parsing with accompanying
evaluation metrics. AMR parsers now obtain high scores on the standard AMR
evaluation metric Smatch, close to or even above reported inter-annotator
agreement. But that does not mean that AMR parsing is solved; in fact, human
evaluation in previous work indicates that current parsers still quite
frequently make errors on node labels or graph structure that substantially
distort sentence meaning. Here, we provide an evaluation suite that tests AMR
parsers on a range of phenomena of practical, technical, and linguistic
interest. Our 36 categories range from seen and unseen labels, to structural
generalization, to coreference. GrAPES reveals in depth the abilities and
shortcomings of current AMR parsers.

我们提出了 Granular AMR Parsing Evaluation Suite (GrAPES)，这是一个用于 Abstract Meaning Representation (AMR) 解析的挑战集合，伴随着相应的评估指标。我们的评估套件测试了 AMR 解析器对实际、技术和语言学上感兴趣的一系列现象。

AMR 解析远未解决：GrAPES，AMR 解析评估套件

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing  Evaluation Suite

Different metrics have been proposed to compare Abstract Meaning
Representation (AMR) graphs. The canonical Smatch metric (Cai and Knight, 2013)
aligns the variables of two graphs and assesses triple matches. The recent
SemBleu metric (Song and Gildea, 2019) is based on the machine-translation
metric Bleu (Papineni et al., 2002) and increases computational efficiency by
ablating the variable-alignment.
In this paper, i) we establish criteria that enable researchers to perform a
principled assessment of metrics comparing meaning representations like AMR;
ii) we undertake a thorough analysis of Smatch and SemBleu where we show that
the latter exhibits some undesirable properties. For example, it does not
conform to the identity of indiscernibles rule and introduces biases that are
hard to control; iii) we propose a novel metric S$^2$match that is more
benevolent to only very slight meaning deviations and targets the fulfilment of
all established criteria. We assess its suitability and show its advantages
over Smatch and SemBleu.

本文提出了一种新的度量标准 S$^2$match，它可以比较 Abstract Meaning Representation (AMR) graphs 并评估 triple matches，与具有类似功能的 Smatch 和 SemBleu 比较后，S$^2$match 具有更好的性能，能够更好地控制偏差。