BriefGPT.xyz
Oct, 2023
特征学习的光谱条件
A Spectral Condition for Feature Learning
HTML
PDF
Greg Yang, James B. Simon, Jeremy Bernstein
TL;DR
通过扩大神经网络的规模进行特征学习，我们展示了通过标度化权重矩阵和它们的更新的谱范数来实现特征学习，这是与根据Frobenius范数和条目大小进行启发式标度化方法相反的，同时我们的谱标度分析还导致了对最大更新参数化的基本推导，总之，我们旨在为读者提供神经网络特征学习的扎实概念理解。
Abstract
The push to train ever larger
neural networks
has motivated the study of initialization and training at
large network width
. A key challenge is to scale training so that a network's internal representations evolv
→