Neural Machine Translation (NMT) systems have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of NMT systems, which is of paramount importance due to often vast translation demands and real-time requirements, has surprisingly received little attention. In this paper, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art NMT systems. By analyzing the working mechanism and implementation of 1455 public-accessible NMT systems, we observe a fundamental property in NMT systems that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that NMT systems would have to go through enough iterations to satisfy the pre-configured threshold. We present NMTSloth, which develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level, which sufficiently delays the appearance of EOS and forces these inputs to reach the naturally-unreachable threshold. To demonstrate the effectiveness of NMTSloth, we conduct a systematic evaluation on three public-available NMT systems: Google T5, AllenAI WMT14, and Helsinki-NLP translators. Experimental results show that NMTSloth can increase NMT systems' response latency and energy consumption by 85% to 3153% and 86% to 3052%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by NMTSloth significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs).

本文通过分析1455个公共可访问的神经机器翻译系统的工作机理和实现，发现了一个可针对性地对其中的一种根本属性进行操纵以大幅降低计算效率。研究人员提出了一个名为NMTSloth的渐变引导技术，它搜索字符级、标记级和结构级的最小和不易察觉的扰动，并通过延迟End Of Sentence的出现来迫使这些输入达到自然不可达的阈值，从而进行系统的测试和评估。通过针对三种公共可用的NMT系统进行实验，以字符或标记的方式干扰输入句子，能够使NMT系统的响应延迟和能量消耗分别增加85%至3153%和86%至3052%。在真实的移动设备上，通过NMTSloth生成的输入会显著影响电池寿命，使其消耗超过正常输入的30倍以上。

NMTSloth: 理解和测试神经机器翻译系统效率退化