Recently, astonishing advances have been observed in AMR parsing, as measured
by the structural Smatch metric. In fact, today's systems achieve performance
levels that seem to surpass estimates of human inter annotator agreement (IAA).
Therefore, it is unclear how well Smatch (still) relates to human estimates of
parse quality, as in this situation potentially fine-grained errors of similar
weight may impact the AMR's meaning to different degrees.
We conduct an analysis of two popular and strong AMR parsers that --
according to Smatch -- reach quality levels on par with human IAA, and assess
how human quality ratings relate to Smatch and other AMR metrics. Our main
findings are: i) While high Smatch scores indicate otherwise, we find that AMR
parsing is far from being solved: we frequently find structurally small, but
semantically unacceptable errors that substantially distort sentence meaning.
ii) Considering high-performance parsers, better Smatch scores may not
necessarily indicate consistently better parsing quality. To obtain a
meaningful and comprehensive assessment of quality differences of parse(r)s, we
recommend augmenting evaluations with macro statistics, use of additional
metrics, and more human analysis.

本研究对两种 AMR 解析器进行了分析，将其与 Smatch 和其他 AMR 指标以及人类质量评估进行了比较，发现高 Smatch 分数不一定表示一致的解析质量，并且经常出现结构上小的、但语义上不可接受的错误，从而建议使用宏统计、其他指标和更多的人类分析来评估解析器的质量差异。