Neural Machine Translation (NMT) has shown remarkable progress over the past
few years with production systems now being deployed to end-users. One major
drawback of current architectures is that they are expensive to train,
typically requiring days to weeks of GPU time to converge. This makes
exhaustive hyperparameter search, as is commonly done with other neural network
architectures, prohibitively expensive. In this work, we present the first
large-scale analysis of NMT architecture hyperparameters. We report empirical
results and variance numbers for several hundred experimental runs,
corresponding to over 250,000 GPU hours on the standard WMT English to German
translation task. Our experiments lead to novel insights and practical advice
for building and extending NMT architectures. As part of this contribution, we
release an open-source NMT framework that enables researchers to easily
experiment with novel techniques and reproduce state of the art results.

本研究对神经机器翻译（NMT）架构的超参数进行了首次大规模分析，并报告了标准 WMT 英德翻译任务上数百个实验结果和方差数，相当于超过 25 万 GPU 小时。通过这些实验，得出了新的见解和建议，建立了一个开源 NMT 框架，使研究人员可以轻松地实验新技术并重现最先进的结果。

神经机器翻译体系结构的大规模探索

Massive Exploration of Neural Machine Translation Architectures

Neural Machine Translation (NMT), like many other deep learning domains,
typically suffers from over-parameterization, resulting in large storage sizes.
This paper examines three simple magnitude-based pruning schemes to compress
NMT models, namely class-blind, class-uniform, and class-distribution, which
differ in terms of how pruning thresholds are computed for the different
classes of weights in the NMT architecture. We demonstrate the efficacy of
weight pruning as a compression technique for a state-of-the-art NMT system. We
show that an NMT model with over 200 million parameters can be pruned by 40%
with very little performance loss as measured on the WMT'14 English-German
translation task. This sheds light on the distribution of redundancy in the NMT
architecture. Our main result is that with retraining, we can recover and even
surpass the original performance with an 80%-pruned model.

本文探讨了三种基于大小的剪枝模式对 NMT 模型进行压缩，通过实验证明了剪枝技术是一种有效的压缩技术，并且可以通过再训练来恢复或甚至超越原始性能。

通过剪枝压缩神经机器翻译模型

Compression of Neural Machine Translation Models via Pruning

Neural Machine Translation (NMT) has obtained state-of-the art performance
for several language pairs, while only using parallel data for training.
Target-side monolingual data plays an important role in boosting fluency for
phrase-based statistical machine translation, and we investigate the use of
monolingual data for NMT. In contrast to previous work, which combines NMT
models with separately trained language models, we note that encoder-decoder
NMT architectures already have the capacity to learn the same information as a
language model, and we explore strategies to train with monolingual data
without changing the neural network architecture. By pairing monolingual
training data with an automatic back-translation, we can treat it as additional
parallel training data, and we obtain substantial improvements on the WMT 15
task English<->German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task
Turkish->English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We
also show that fine-tuning on in-domain monolingual and parallel data gives
substantial improvements for the IWSLT 15 task English->German.

通过将单语数据与自动背景翻译配对，我们未改变神经网络结构，使用目标语单语训练数据进行神经机器翻译（NMT）模型的训练，并在多项任务上取得最新的最优结果（最高 + 2.8-3.7 BLEU），并证明了使用领域单语和平行数据进行微调，对 IWSLT 15 任务英德翻译有实质性的改善。