Recently, astonishing advances have been observed in AMR parsing, as measured
by the structural Smatch metric. In fact, today's systems achieve performance
levels that seem to surpass estimates of human inter annotator agreement (IAA).
Therefore, it is unclear how well Smatch (still) relates to human estimates of
parse quality, as in this situation potentially fine-grained errors of similar
weight may impact the AMR's meaning to different degrees.
We conduct an analysis of two popular and strong AMR parsers that --
according to Smatch -- reach quality levels on par with human IAA, and assess
how human quality ratings relate to Smatch and other AMR metrics. Our main
findings are: i) While high Smatch scores indicate otherwise, we find that AMR
parsing is far from being solved: we frequently find structurally small, but
semantically unacceptable errors that substantially distort sentence meaning.
ii) Considering high-performance parsers, better Smatch scores may not
necessarily indicate consistently better parsing quality. To obtain a
meaningful and comprehensive assessment of quality differences of parse(r)s, we
recommend augmenting evaluations with macro statistics, use of additional
metrics, and more human analysis.

本研究对两种 AMR 解析器进行了分析，将其与 Smatch 和其他 AMR 指标以及人类质量评估进行了比较，发现高 Smatch 分数不一定表示一致的解析质量，并且经常出现结构上小的、但语义上不可接受的错误，从而建议使用宏统计、其他指标和更多的人类分析来评估解析器的质量差异。

更好的 Smatch = 更好的解析器？AMR 评估不再那么简单

Better Smatch = Better Parser? AMR evaluation is not so simple anymore

We present a novel technique for automatic program correction in MOOCs,
capable of fixing both syntactic and semantic errors without manual, problem
specific correction strategies. Given an incorrect student program, it
generates candidate programs from a distribution of likely corrections, and
checks each candidate for correctness against a test suite.
The key observation is that in MOOCs many programs share similar code
fragments, and the seq2seq neural network model, used in the natural-language
processing task of machine translation, can be modified and trained to recover
these fragments.
Experiment shows our scheme can correct 29% of all incorrect submissions and
out-performs state of the art approach which requires manual, problem specific
correction strategies.

本文提出了一种用于自动纠正 MOOCs 中程序错误的新技术，能够修复语法和语义错误而不需要手动进行问题特定的纠正策略，实验表明本方案可以更有效地修正程序错误。