Entropy and mutual information in neural networks provide rich information on
the learning process, but they have proven difficult to compute reliably in
high dimensions. Indeed, in noisy and high-dimensional data, traditional
estimates in ambient dimensions approach a fixed entropy and are prohibitively
hard to compute. To address these issues, we leverage data geometry to access
the underlying manifold and reliably compute these information-theoretic
measures. Specifically, we define diffusion spectral entropy (DSE) in neural
representations of a dataset as well as diffusion spectral mutual information
(DSMI) between different variables representing data. First, we show that they
form noise-resistant measures of intrinsic dimensionality and relationship
strength in high-dimensional simulated data that outperform classic Shannon
entropy, nonparametric estimation, and mutual information neural estimation
(MINE). We then study the evolution of representations in classification
networks with supervised learning, self-supervision, or overfitting. We observe
that (1) DSE of neural representations increases during training; (2) DSMI with
the class label increases during generalizable learning but stays stagnant
during overfitting; (3) DSMI with the input signal shows differing trends: on
MNIST it increases, while on CIFAR-10 and STL-10 it decreases. Finally, we show
that DSE can be used to guide better network initialization and that DSMI can
be used to predict downstream classification accuracy across 962 models on
ImageNet. The official implementation is available at
this https URL

我们通过利用数据几何形态来计算熵和互信息，从而可靠地评估高维数据中的学习过程，证明其在分类网络中的增长和影响。

使用抗噪扩散谱熵评估神经网络训练过程中的表示

Assessing Neural Network Representations During Training Using  Noise-Resilient Diffusion Spectral Entropy

Continual learning aims to efficiently learn from a non-stationary stream of
data while avoiding forgetting the knowledge of old data. In many practical
applications, data complies with non-Euclidean geometry. As such, the commonly
used Euclidean space cannot gracefully capture non-Euclidean geometric
structures of data, leading to inferior results. In this paper, we study
continual learning from a novel perspective by exploring data geometry for the
non-stationary stream of data. Our method dynamically expands the geometry of
the underlying space to match growing geometric structures induced by new data,
and prevents forgetting by keeping geometric structures of old data into
account. In doing so, making use of the mixed curvature space, we propose an
incremental search scheme, through which the growing geometric structures are
encoded. Then, we introduce an angular-regularization loss and a
neighbor-robustness loss to train the model, capable of penalizing the change
of global geometric structures and local geometric structures. Experiments show
that our method achieves better performance than baseline methods designed in
Euclidean space.

本文从数据几何的角度出发，研究了在不断变化的数据流中的持续学习问题。研究者提出了一种动态扩展底层空间几何结构的方法，配合使用各向异性空间与混合曲率空间对增长的几何结构进行编码。同时，引入了角度正则化损失与近邻抗扰损失的训练方式，在防止旧数据遗忘的前提下，取得了比欧氏空间方案更好的表现。

探索数据几何在连续学习中的应用

Exploring Data Geometry for Continual Learning

Knowledge distillation, i.e., one classifier being trained on the outputs of
another classifier, is an empirically very successful technique for knowledge
transfer between classifiers. It has even been observed that classifiers learn
much faster and more reliably if trained with the outputs of another classifier
as soft labels, instead of from ground truth data. So far, however, there is no
satisfactory theoretical explanation of this phenomenon. In this work, we
provide the first insights into the working mechanisms of distillation by
studying the special case of linear and deep linear classifiers. Specifically,
we prove a generalization bound that establishes fast convergence of the
expected risk of a distillation-trained linear classifier. From the bound and
its proof we extract three key factors that determine the success of
distillation: * data geometry -- geometric properties of the data distribution,
in particular class separation, has a direct influence on the convergence speed
of the risk; * optimization bias -- gradient descent optimization finds a very
favorable minimum of the distillation objective; and * strong monotonicity --
the expected risk of the student classifier always decreases when the size of
the training set grows.

本研究通过研究线性和深度线性分类器的特殊情况，证明了知识蒸馏在理论上的有效性，并揭示了决定其成功的三个关键因素：数据几何形态、优化偏差和强单调性。