In this paper, we examine the current state-of-the-art in AMR parsing, which
relies on ensemble strategies by merging multiple graph predictions. Our
analysis reveals that the present models often violate AMR structural
constraints. To address this issue, we develop a validation method, and show
how ensemble models can exploit SMATCH metric weaknesses to obtain higher
scores, but sometimes result in corrupted graphs. Additionally, we highlight
the demanding need to compute the SMATCH score among all possible predictions.
To overcome these challenges, we propose two novel ensemble strategies based on
Transformer models, improving robustness to structural constraints, while also
reducing the computational time. Our methods provide new insights for enhancing
AMR parsers and metrics. Our code is available at
\href{this https URL}{github.com/babelscape/AMRs-Assemble}.

本文研究了 AMR 解析的现状和问题，提出了两种基于 Transformer 模型的集成策略来解决结构约束问题，并强调了计算所有可能预测的 SMATCH 得分的需求。

AMRs 统合！使用自回归模型学习进行 AMR 解析统合

AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR  Parsing

The Smatch metric is a popular method for evaluating graph distances, as is
necessary, for instance, to assess the performance of semantic graph parsing
systems. However, we observe some issues in the metric that jeopardize
meaningful evaluation. E.g., opaque pre-processing choices can affect results,
and current graph-alignment solvers do not provide us with upper-bounds.
Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore,
adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity)
are spread out, and lack a unifying framework.
For better inspection, we divide the metric into three modules:
pre-processing, alignment, and scoring. Examining each module, we specify its
goals and diagnose potential issues, for which we discuss and test mitigation
strategies. For pre-processing, we show how to fully conform to annotation
guidelines that allow structurally deviating but valid graphs. For safer and
enhanced alignment, we show the feasibility of optimal alignment in a standard
evaluation setup, and develop a lossless graph compression method that shrinks
the search space and significantly increases efficiency. For improved scoring,
we propose standardized and extended metric calculation of fine-grained
sub-graph meaning aspects. Our code is available at
this https URL

该文介绍了 Smach 指标存在的问题，将其分为三个模块，分别为 pre-processing, alignment 和 scoring, 并分别提出应对策略，最终实现了对语义图解析系统的更高效和更公平的评估。

SMATCH++：语义图标准化和扩展评估

SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

Recently, astonishing advances have been observed in AMR parsing, as measured
by the structural Smatch metric. In fact, today's systems achieve performance
levels that seem to surpass estimates of human inter annotator agreement (IAA).
Therefore, it is unclear how well Smatch (still) relates to human estimates of
parse quality, as in this situation potentially fine-grained errors of similar
weight may impact the AMR's meaning to different degrees.
We conduct an analysis of two popular and strong AMR parsers that --
according to Smatch -- reach quality levels on par with human IAA, and assess
how human quality ratings relate to Smatch and other AMR metrics. Our main
findings are: i) While high Smatch scores indicate otherwise, we find that AMR
parsing is far from being solved: we frequently find structurally small, but
semantically unacceptable errors that substantially distort sentence meaning.
ii) Considering high-performance parsers, better Smatch scores may not
necessarily indicate consistently better parsing quality. To obtain a
meaningful and comprehensive assessment of quality differences of parse(r)s, we
recommend augmenting evaluations with macro statistics, use of additional
metrics, and more human analysis.

本研究对两种 AMR 解析器进行了分析，将其与 Smatch 和其他 AMR 指标以及人类质量评估进行了比较，发现高 Smatch 分数不一定表示一致的解析质量，并且经常出现结构上小的、但语义上不可接受的错误，从而建议使用宏统计、其他指标和更多的人类分析来评估解析器的质量差异。