We present GenEFT: an effective theory framework for shedding light on the
statics and dynamics of neural network generalization, and illustrate it with
graph learning examples. We first investigate the generalization phase
transition as data size increases, comparing experimental results with
information-theory-based approximations. We find generalization in a Goldilocks
zone where the decoder is neither too weak nor too powerful. We then introduce
an effective theory for the dynamics of representation learning, where
latent-space representations are modeled as interacting particles (repons), and
find that it explains our experimentally observed phase transition between
generalization and overfitting as encoder and decoder learning rates are
scanned. This highlights the power of physics-inspired effective theories for
bridging the gap between theoretical predictions and practice in machine
learning.

我们提出了 GenEFT：一种有效的理论框架来阐明神经网络泛化的静态和动态，并通过图学习的示例进行说明。我们首先研究了数据大小增加时的泛化相变，并将实验结果与基于信息论的近似进行了比较，发现在解码器既不太弱也不太强的 “金发区” 中存在泛化。然后，我们引入了一种表示学习动态的有效理论，将潜空间表示建模为相互作用的粒子（repons），并发现它能够解释我们实验观察到的泛化与过拟合之间的相变，而编码器和解码器的学习率被扫描。这突显了受物理学启发的有效理论在理论预测与机器学习实践之间弥合差距的能力。

GenEFT：通过有效理论理解模型泛化的静态和动态

GenEFT: Understanding Statics and Dynamics of Model Generalization via  Effective Theory

The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical
variants have been proposed as a proxy to capture certain behaviors of real
neural networks. In this work, we study NTKs through the lens of scaling laws,
and demonstrate that they fall short of explaining important aspects of neural
network generalization. In particular, we demonstrate realistic settings where
finite-width neural networks have significantly better data scaling exponents
as compared to their corresponding empirical and infinite NTKs at
initialization. This reveals a more fundamental difference between the real
networks and NTKs, beyond just a few percentage points of test accuracy.
Further, we show that even if the empirical NTK is allowed to be pre-trained on
a constant number of samples, the kernel scaling does not catch up to the
neural network scaling. Finally, we show that the empirical NTK continues to
evolve throughout most of the training, in contrast with prior work which
suggests that it stabilizes after a few epochs of training. Altogether, our
work establishes concrete limitations of the NTK approach in understanding
generalization of real networks on natural datasets.

本文通过缩放定律的角度研究神经切向核 (NTK) 及其经验性变量，发现它们无法完全解释神经网络泛化的重要方面。通过实际设置，我们展示了有限宽度神经网络相对于其对应的经验和无穷 NTK 起始时具有显着更好的数据缩放指数，并证明了 NTK 方法在理解自然数据集上真实网络泛化的局限性。

深度学习中 NTK 在理解泛化方面的局限性

Limitations of the NTK for Understanding Generalization in Deep Learning

In this paper, we propose a deep convolutional recurrent neural network that
predicts action sequences for task and motion planning (TAMP) from an initial
scene image. Typical TAMP problems are formalized by combining reasoning on a
symbolic, discrete level (e.g. first-order logic) with continuous motion
planning such as nonlinear trajectory optimization. Due to the great
combinatorial complexity of possible discrete action sequences, a large number
of optimization/motion planning problems have to be solved to find a solution,
which limits the scalability of these approaches.
To circumvent this combinatorial complexity, we develop a neural network
which, based on an initial image of the scene, directly predicts promising
discrete action sequences such that ideally only one motion planning problem
has to be solved to find a solution to the overall TAMP problem. A key aspect
is that our method generalizes to scenes with many and varying number of
objects, although being trained on only two objects at a time. This is possible
by encoding the objects of the scene in images as input to the neural network,
instead of a fixed feature vector. Results show runtime improvements of several
magnitudes. Video: this https URL

本文提出了一种基于场景图像的深度卷积循环神经网络，可用于从一个初始化的场景图像中预测任务与运动规划中的动作序列。该网络可避免组合复杂性并具有很好的泛化性能，可加速任务与运动规划的运行时间。