We propose a new method to probe the learning mechanism of Deep Neural
Networks (DNN) by perturbing the system using Noise Injection Nodes (NINs).
These nodes inject uncorrelated noise via additional optimizable weights to
existing feed-forward network architectures, without changing the optimization
algorithm. We find that the system displays distinct phases during training,
dictated by the scale of injected noise. We first derive expressions for the
dynamics of the network and utilize a simple linear model as a test case. We
find that in some cases, the evolution of the noise nodes is similar to that of
the unperturbed loss, thus indicating the possibility of using NINs to learn
more about the full system in the future.

我们提出了一种新方法，通过使用噪声注入节点来干扰系统来探究深度神经网络（DNN）的学习机制。这些节点通过额外的可优化权重注入不相关噪声到现有的前馈网络架构中，而不改变优化算法。我们发现，系统在训练期间显示出不同的阶段，这些阶段由注入噪声的规模决定。我们首先推导出网络动力学的表达式，并将一个简单的线性模型作为测试用例。我们发现，在某些情况下，噪声节点的演变类似于未受干扰的损失，因此指示着将来能够使用噪声注入节点来更好地了解整个系统的可能性。