We reassess the claims of human parity and super-human performance made at
the news shared task of WMT 2019 for three translation directions:
English-to-German, English-to-Russian and German-to-English. First we identify
three potential issues in the human evaluation of that shared task: (i) the
limited amount of intersentential context available, (ii) the limited
translation proficiency of the evaluators and (iii) the use of a reference
translation. We then conduct a modified evaluation taking these issues into
account. Our results indicate that all the claims of human parity and
super-human performance made at WMT 2019 should be refuted, except the claim of
human parity for English-to-German. Based on our findings, we put forward a set
of recommendations and open questions for future assessments of human parity in
machine translation.

本文重新评估 WMT 2019 机器翻译任务中英德、英俄和德英三个方向人工翻译水平与超人类水平的声明，发现除了英德方向人类翻译水平与机器翻译持平外，其他所有声明都被推翻，并提出了启示和未来评估的开放性问题。

WMT 2019 中机器翻译人类水平与超人表现的重新评估

Reassessing Claims of Human Parity and Super-Human Performance in  Machine Translation at WMT 2019

We reassess a recent study (Hassan et al., 2018) that claimed that machine
translation (MT) has reached human parity for the translation of news from
Chinese into English, using pairwise ranking and considering three variables
that were not taken into account in that previous study: the language in which
the source side of the test set was originally written, the translation
proficiency of the evaluators, and the provision of inter-sentential context.
If we consider only original source text (i.e. not translated from another
language, or translationese), then we find evidence showing that human parity
has not been achieved. We compare the judgments of professional translators
against those of non-experts and discover that those of the experts result in
higher inter-annotator agreement and better discrimination between human and
machine translations. In addition, we analyse the human translations of the
test set and identify important translation issues. Finally, based on these
findings, we provide a set of recommendations for future human evaluations of
MT.

本文重新评估了一项最近的研究 (Hassan et al.,2018)，即使用成对排序法并考虑到三个变量，重申机器翻译在从中文到英文的新闻翻译方面已达到人类水平，我们提供了一组人类评估未来机器翻译的建议。

人工神经网络翻译中人类表现的重新评估：实现不可实现的目标？

Attaining the Unattainable? Reassessing Claims of Human Parity in Neural  Machine Translation

Machine translation has made rapid advances in recent years. Millions of
people are using it today in online translation systems and mobile applications
in order to communicate across language barriers. The question naturally arises
whether such systems can approach or achieve parity with human translations. In
this paper, we first address the problem of how to define and accurately
measure human parity in translation. We then describe Microsoft's machine
translation system and measure the quality of its translations on the widely
used WMT 2017 news translation task from Chinese to English. We find that our
latest neural machine translation system has reached a new state-of-the-art,
and that the translation quality is at human parity when compared to
professional human translations. We also find that it significantly exceeds the
quality of crowd-sourced non-professional translations.

该篇论文讨论了机器翻译的发展与挑战，评估了微软近期机器翻译系统在中英新闻翻译中的表现，结果显示该系统已经能够达到人类专业翻译水平。

自动中英文新闻翻译达到人类水平

Achieving Human Parity on Automatic Chinese to English News Translation

Conversational speech recognition has served as a flagship speech recognition
task since the release of the Switchboard corpus in the 1990s. In this paper,
we measure the human error rate on the widely used NIST 2000 test set, and find
that our latest automated system has reached human parity. The error rate of
professional transcribers is 5.9% for the Switchboard portion of the data, in
which newly acquainted pairs of people discuss an assigned topic, and 11.3% for
the CallHome portion where friends and family members have open-ended
conversations. In both cases, our automated system establishes a new state of
the art, and edges past the human benchmark, achieving error rates of 5.8% and
11.0%, respectively. The key to our system's performance is the use of various
convolutional and LSTM acoustic model architectures, combined with a novel
spatial smoothing method and lattice-free MMI acoustic training, multiple
recurrent neural network language modeling approaches, and a systematic use of
system combination.

本篇论文测试了 Convolutional 和 LSTM acoustic model architectures 在对话语音识别上的效果，并采用 novel spatial smoothing method 和 lattice-free MMI acoustic training 方法，还有 multiple recurrent neural network language modeling approaches 以及 system combination 等方法，实现了与人类专业转录员相媲美的识别准确率。