We introduce Natural Neural Networks, a novel family of algorithms that speed
up convergence by adapting their internal representation during training to
improve conditioning of the Fisher matrix. In particular, we show a specific
example that employs a simple and efficient reparametrization of the neural
network weights by implicitly whitening the representation obtained at each
layer, while preserving the feed-forward computation of the network. Such
networks can be trained efficiently via the proposed Projected Natural Gradient
Descent algorithm (PRONG), which amortizes the cost of these reparametrizations
over many parameter updates and is closely related to the Mirror Descent online
learning algorithm. We highlight the benefits of our method on both
unsupervised and supervised learning tasks, and showcase its scalability by
training on the large-scale ImageNet Challenge dataset.

研究引入了自然神经网络，旨在通过调整内部表示来加速收敛并改善 Fisher 矩阵的条件。使用简单高效的参数重整方式，从而能够保留神经网络的正向传播计算。研究表明，通过提出的投影自然梯度下降算法 (PRONG)，这种神经网络能够高效地训练；研究同时在无监督和监督学习任务上证明了该方法的优点，并通过对大规模 ImageNet Challenge 数据集的训练显示了其可伸缩性。