We develop information-geometric techniques to analyze the trajectories of
the predictions of deep networks during training. By examining the underlying
high-dimensional probabilistic models, we reveal that the training process
explores an effectively low-dimensional manifold. Networks with a wide range of
architectures, sizes, trained using different optimization methods,
regularization techniques, data augmentation techniques, and weight
initializations lie on the same manifold in the prediction space. We study the
details of this manifold to find that networks with different architectures
follow distinguishable trajectories but other factors have a minimal influence;
larger networks train along a similar manifold as that of smaller networks,
just faster; and networks initialized at very different parts of the prediction
space converge to the solution along a similar manifold.

通过信息几何技术，分析深度网络在训练过程中的预测轨迹，揭示了网络训练过程中探索了一个有效低维度流形，在预测空间中，不同架构、大小、训练优化方法、正则化技术、数据增强技术和权重初始化的网络都在同一个流形上，并研究了该流形的细节，发现不同的网络架构遵循可区分的轨迹，但其他因素影响较小；较大的网络沿着较小的网络相似的流形进行训练，而且更快速；初始化于预测空间不同部分的网络也沿着相似的流形收敛到解决方案。

许多深层网络的训练过程可以被解释为探索同一低维流形

The Training Process of Many Deep Networks Explores the Same  Low-Dimensional Manifold

The proliferation of automated inference algorithms in Bayesian statistics
has provided practitioners newfound access to fast, reproducible data analysis
and powerful statistical models. Designing automated methods that are also both
computationally scalable and theoretically sound, however, remains a
significant challenge. Recent work on Bayesian coresets takes the approach of
compressing the dataset before running a standard inference algorithm,
providing both scalability and guarantees on posterior approximation error. But
the automation of past coreset methods is limited because they depend on the
availability of a reasonable coarse posterior approximation, which is difficult
to specify in practice. In the present work we remove this requirement by
formulating coreset construction as sparsity-constrained variational inference
within an exponential family. This perspective leads to a novel construction
via greedy optimization, and also provides a unifying information-geometric
view of present and past methods. The proposed Riemannian coreset construction
algorithm is fully automated, requiring no problem-specific inputs aside from
the probabilistic model and dataset. In addition to being significantly easier
to use than past methods, experiments demonstrate that past coreset
constructions are fundamentally limited by the fixed coarse posterior
approximation; in contrast, the proposed algorithm is able to continually
improve the coreset, providing state-of-the-art Bayesian dataset summarization
with orders-of-magnitude reduction in KL divergence to the exact posterior.

本研究提出了一种基于稀疏约束变分推断视角的 Riemannian coresets 构建算法，与过去的方法相比，该算法不需要一个合理的后验近似。实验结果表明，提出的算法能够不断改善 coreset，大大减小 KL 散度，从而提供最先进的 Bayesian 数据集概括。