We study how the topology of feature embedding space changes as it passes
through the layers of a well-trained deep neural network (DNN) through Betti
numbers. Motivated by existing studies using simplicial complexes on shallow
fully connected networks (FCN), we present an extended analysis using Cubical
homology instead, with a variety of popular deep architectures and real image
datasets. We demonstrate that as depth increases, a topologically complicated
dataset is transformed into a simple one, resulting in Betti numbers attaining
their lowest possible value. The rate of decay in topological complexity (as a
metric) helps quantify the impact of architectural choices on the
generalization ability. Interestingly from a representation learning
perspective, we highlight several invariances such as topological invariance of
(1) an architecture on similar datasets; (2) embedding space of a dataset for
architectures of variable depth; (3) embedding space to input resolution/size,
and (4) data sub-sampling. In order to further demonstrate the link between
expressivity \& the generalization capability of a network, we consider the
task of ranking pre-trained models for downstream classification task (transfer
learning). Compared to existing approaches, the proposed metric has a better
correlation to the actually achievable accuracy via fine-tuning the pre-trained
model.

通过 Betti 数我们研究了在经过深度神经网络的各个层时，特征嵌入空间的拓扑结构如何变化。我们使用了拓扑同调理论中的方格同调进行了扩展分析，使用了各种流行的深度架构和真实图像数据集。我们证明随着深度的增加，一个拓扑上复杂的数据集会被转换成一个简单的数据集，Betti 数会取得最低可能的值。拓扑复杂度的衰减速率可以量化架构选择对泛化能力的影响。此外，我们从表示学习的角度强调了几种不变性，例如 (1) 相似数据集上的体系结构、(2) 深度可变的嵌入空间、(3) 嵌入空间与输入分辨率 / 大小以及 (4) 数据子采样。为了进一步证明网络的表达能力与泛化能力之间的联系，我们考虑了下游分类任务 (迁移学习) 中预训练模型的排序任务。与现有方法相比，所提出的度量方法与通过微调预训练模型实际可达到的准确性具有更好的相关性。

利用代数拓扑描述神经网络嵌入空间的演化

On Characterizing the Evolution of Embedding Space of Neural Networks  using Algebraic Topology

Physics-informed neural networks (PINNs) have been popularized as a deep
learning framework that can seamlessly synthesize observational data and
partial differential equation (PDE) constraints. Their practical effectiveness
however can be hampered by training pathologies, but also oftentimes by poor
choices made by users who lack deep learning expertise. In this paper we
present a series of best practices that can significantly improve the training
efficiency and overall accuracy of PINNs. We also put forth a series of
challenging benchmark problems that highlight some of the most prominent
difficulties in training PINNs, and present comprehensive and fully
reproducible ablation studies that demonstrate how different architecture
choices and training strategies affect the test accuracy of the resulting
models. We show that the methods and guiding principles put forth in this study
lead to state-of-the-art results and provide strong baselines that future
studies should use for comparison purposes. To this end, we also release a
highly optimized library in JAX that can be used to reproduce all results
reported in this paper, enable future research studies, as well as facilitate
easy adaptation to new use-case scenarios.

通过提出一系列最佳实践，改进物理信息神经网络（PINNs）的训练效率和整体准确性，还展示了不同架构选择和训练策略如何影响结果模型的测试准确性。