Despite the great success of NMT, there still remains a severe challenge: it
is hard to interpret the internal dynamics during its training process. In this
paper we propose to understand learning dynamics of NMT by using a recent
proposed technique named Loss Change Allocation
(LCA)~\citep{lan-2019-loss-change-allocation}. As LCA requires calculating the
gradient on an entire dataset for each update, we instead present an
approximate to put it into practice in NMT scenario. %motivated by the lesson
from sgd. Our simulated experiment shows that such approximate calculation is
efficient and is empirically proved to deliver consistent results to the
brute-force implementation. In particular, extensive experiments on two
standard translation benchmark datasets reveal some valuable findings.

该研究旨在通过使用最新提出的技术 LCA 了解 NMT 的学习动态，提出了一种在 NMT 场景中实现 LCA 的近似方法，模拟实验证明此方法是有效的并能够在两个标准的翻译基准数据集上提供有价值的发现。

神经机器翻译学习动态的理解

Understanding Learning Dynamics for Neural Machine Translation

Neural networks enjoy widespread use, but many aspects of their training,
representation, and operation are poorly understood. In particular, our view
into the training process is limited, with a single scalar loss being the most
common viewport into this high-dimensional, dynamic process. We propose a new
window into training called Loss Change Allocation (LCA), in which credit for
changes to the network loss is conservatively partitioned to the parameters.
This measurement is accomplished by decomposing the components of an
approximate path integral along the training trajectory using a Runge-Kutta
integrator. This rich view shows which parameters are responsible for
decreasing or increasing the loss during training, or which parameters "help"
or "hurt" the network's learning, respectively. LCA may be summed over training
iterations and/or over neurons, channels, or layers for increasingly coarse
views. This new measurement device produces several insights into training. (1)
We find that barely over 50% of parameters help during any given iteration. (2)
Some entire layers hurt overall, moving on average against the training
gradient, a phenomenon we hypothesize may be due to phase lag in an oscillatory
training process. (3) Finally, increments in learning proceed in a synchronized
manner across layers, often peaking on identical iterations.

提出一种名为 LCA 的新视窗，通过使用 Runge-Kutta 积分器沿训练轨迹分解近似路径积分的组成部分，将网络损失变化的信用保守地分配给参数，从而显示哪些参数在训练过程中负责减少或增加损失，或者哪些参数在网络学习中 “有益” 或 “有害”。该新的测量设备对训练产生了几个不同见解。