Machine translation systems have been widely adopted in our daily life,
making life easier and more convenient. Unfortunately, erroneous translations
may result in severe consequences, such as financial losses. This requires to
improve the accuracy and the reliability of machine translation systems.
However, it is challenging to test machine translation systems because of the
complexity and intractability of the underlying neural models. To tackle these
challenges, we propose a novel metamorphic testing approach by syntactic tree
pruning (STP) to validate machine translation systems. Our key insight is that
a pruned sentence should have similar crucial semantics compared with the
original sentence. Specifically, STP (1) proposes a core semantics-preserving
pruning strategy by basic sentence structure and dependency relations on the
level of syntactic tree representation; (2) generates source sentence pairs
based on the metamorphic relation; (3) reports suspicious issues whose
translations break the consistency property by a bag-of-words model. We further
evaluate STP on two state-of-the-art machine translation systems (i.e., Google
Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs.
The results show that STP can accurately find 5,073 unique erroneous
translations in Google Translate and 5,100 unique erroneous translations in
Bing Microsoft Translator (400% more than state-of-the-art techniques), with
64.5% and 65.4% precision, respectively. The reported erroneous translations
vary in types and more than 90% of them cannot be found by state-of-the-art
techniques. There are 9,393 erroneous translations unique to STP, which is
711.9% more than state-of-the-art techniques. Moreover, STP is quite effective
to detect translation errors for the original sentences with a recall reaching
74.0%, improving state-of-the-art techniques by 55.1% on average.

通过基于语法树修剪的新型变异测试方法，验证机器翻译系统以提高准确性和可靠性；在 Google 翻译和 Bing 微软翻译中准确检测出 5073 和 5100 种错误翻译，比现有技术增加了 400%，错误翻译类型多样且大部分现有技术无法捕获，而新方法对原始句子的检测召回率达到 74.0%，平均比现有技术提高了 55.1%。