Our study reveals new theoretical insights into over-smoothing and feature
over-correlation in deep graph neural networks. We show the prevalence of
invariant subspaces, demonstrating a fixed relative behavior that is unaffected
by feature transformations. Our work clarifies recent observations related to
convergence to a constant state and a potential over-separation of node states,
as the amplification of subspaces only depends on the spectrum of the
aggregation function. In linear scenarios, this leads to node representations
being dominated by a low-dimensional subspace with an asymptotic convergence
rate independent of the feature transformations. This causes a rank collapse of
the node representations, resulting in over-smoothing when smooth vectors span
this subspace, and over-correlation even when over-smoothing is avoided. Guided
by our theory, we propose a sum of Kronecker products as a beneficial property
that can provably prevent over-smoothing, over-correlation, and rank collapse.
We empirically extend our insights to the non-linear case, demonstrating the
inability of existing models to capture linearly independent features.

我们的研究揭示了深度图神经网络中的超平滑和特征过相关性的新的理论洞见。我们展示了不变子空间的普遍性，证明了固定的相对行为不受特征变换的影响。在线性场景中，这导致节点表示被低维子空间主导，其渐近收敛速率与特征变换无关，这导致节点表示的秩崩溃，从而在平滑向量跨越该子空间时产生超平滑，在避免超平滑的情况下产生超相关。根据我们的理论，我们提出了作为有益属性的 Kronecker 乘积之和，可以明确防止超平滑、超相关和秩崩溃。我们从实证角度将我们的洞见扩展到非线性情况，证明了现有模型无法捕捉线性独立特征。

图神经网络中的排名崩溃导致过度平滑和过度相关

Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural  Networks

In this work, we study non-asymptotic bounds on correlation between two time
realizations of stable linear systems with isotropic Gaussian noise.
Consequently, via sampling from a sub-trajectory and using \emph{Talagrands'}
inequality, we show that empirical averages of reward concentrate around steady
state (dynamical system mixes to when closed loop system is stable under linear
feedback policy ) reward , with high-probability. As opposed to common belief
of larger the spectral radius stronger the correlation between samples,
\emph{large discrepancy between algebraic and geometric multiplicity of system
eigenvalues leads to large invariant subspaces related to system-transition
matrix}; once the system enters the large invariant subspace it will travel
away from origin for a while before coming close to a unit ball centered at
origin where an isotropic Gaussian noise can with high probability allow it to
escape the current invariant subspace it resides in, leading to
\emph{bottlenecks} between different invariant subspaces that span
$\mathbb{R}^{n}$, to be precise : system initiated in a large invariant
subspace will be stuck there for a long-time: log-linear in dimension of the
invariant subspace and inversely to log of inverse of magnitude of the
eigenvalue. In the problem of Ordinary Least Squares estimate of system
transition matrix via a single trajectory, this phenomenon is even more evident
if spectrum of transition matrix associated to large invariant subspace is
explosive and small invariant subspaces correspond to stable eigenvalues. Our
analysis provide first interpretable and geometric explanation into intricacies
of learning and concentration for random dynamical systems on continuous, high
dimensional state space; exposing us to surprises in high dimensions

研究了高维状态空间中的非渐近稳定性随机动力系统，通过采样子轨迹和利用 Talagrand 的不等式，证明了奖励的经验均值集中于稳态回报，探讨了系统的不变子空间之间的瓶颈现象以及及其对随机动力系统的学习和集中性的影响。

高维线性高斯模型的学习与集中：一种不变子空间方法

Learning and Concentration for High Dimensional Linear Gaussians: an  Invariant Subspace Approach

Spectral decomposition of the Koopman operator is attracting attention as a
tool for the analysis of nonlinear dynamical systems. Dynamic mode
decomposition is a popular numerical algorithm for Koopman spectral analysis;
however, we often need to prepare nonlinear observables manually according to
the underlying dynamics, which is not always possible since we may not have any
a priori knowledge about them. In this paper, we propose a fully data-driven
method for Koopman spectral analysis based on the principle of learning Koopman
invariant subspaces from observed data. To this end, we propose minimization of
the residual sum of squares of linear least-squares regression to estimate a
set of functions that transforms data into a form in which the linear
regression fits well. We introduce an implementation with neural networks and
evaluate performance empirically using nonlinear dynamical systems and
applications.

该研究提出了一种基于数据驱动的 Koopman 谱分析方法，该方法通过最小化线性最小二乘回归的残差平方和来估计一组将数据转换为线性回归适合的形式的函数，并采用神经网络实现并在非线性动力系统及其应用中进行了实证评估。