Neural networks have become a widely adopted tool for tackling a variety of
problems in machine learning and artificial intelligence. In this contribution
we use the mathematical framework of local stability analysis to gain a deeper
understanding of the learning dynamics of feed forward neural networks.
Therefore, we derive equations for the tangent operator of the learning
dynamics of three-layer networks learning regression tasks. The results are
valid for an arbitrary numbers of nodes and arbitrary choices of activation
functions. Applying the results to a network learning a regression task, we
investigate numerically, how stability indicators relate to the final
training-loss. Although the specific results vary with different choices of
initial conditions and activation functions, we demonstrate that it is possible
to predict the final training loss, by monitoring finite-time Lyapunov
exponents or covariant Lyapunov vectors during the training process.

使用局部稳定性分析的数学框架，我们研究了前馈神经网络学习动力学的深层理解，推导了三层神经网络在学习回归任务时的切线算子方程，结果适用于任意节点数和任意激活函数的选择。我们通过数值方法应用这些结果于网络学习回归任务中，调查了稳定性指标与最终训练损失之间的关系。虽然具体结果会因初始条件和激活函数的不同而有所变化，我们证明了通过监测训练过程中的有限时间 Lyapunov 指数或协变 Lyapunov 向量，可以预测最终的训练损失。

学习网络的权重动态

On the weight dynamics of learning networks

High-dimensional datasets present substantial challenges in statistical
modeling across various disciplines, necessitating effective dimensionality
reduction methods. Deep learning approaches, notable for their capacity to
distill essential features from complex data, facilitate modeling,
visualization, and compression through reduced dimensionality latent feature
spaces, have wide applications from bioinformatics to earth sciences. This
study introduces a novel workflow to evaluate the stability of these latent
spaces, ensuring consistency and reliability in subsequent analyses. Stability,
defined as the invariance of latent spaces to minor data, training
realizations, and parameter perturbations, is crucial yet often overlooked.
Our proposed methodology delineates three stability types, sample,
structural, and inferential, within latent spaces, and introduces a suite of
metrics for comprehensive evaluation. We implement this workflow across 500
autoencoder realizations and three datasets, encompassing both synthetic and
real-world scenarios to explain latent space dynamics. Employing k-means
clustering and the modified Jonker-Volgenant algorithm for class alignment,
alongside anisotropy metrics and convex hull analysis, we introduce adjusted
stress and Jaccard dissimilarity as novel stability indicators.
Our findings highlight inherent instabilities in latent feature spaces and
demonstrate the workflow's efficacy in quantifying and interpreting these
instabilities. This work advances the understanding of latent feature spaces,
promoting improved model interpretability and quality control for more informed
decision-making for diverse analytical workflows that leverage deep learning.

高维数据集在各个学科的统计建模中带来了很大的挑战，需要高效的降维方法。深度学习方法通过降维的潜在特征空间从复杂数据中提取关键特征，有着广泛的应用，从生物信息学到地球科学等领域。本研究通过引入一个新的工作流程来评估这些潜在空间的稳定性，确保后续分析的一致性和可靠性。该工作流程囊括了三种稳定性类型：样本稳定性、结构稳定性和推断稳定性，并引入了一套综合评估指标。研究结果揭示了潜在特征空间固有的不稳定性，并证明了该工作流程在量化和解释这些不稳定性方面的功效。这项工作推进了对潜在特征空间的理解，促进了深度学习在各种分析工作流中的模型可解释性和质量控制，为更加明智的决策提供了基础。