Bounding and predicting the generalization gap of overparameterized neural
networks remains a central open problem in theoretical machine learning. Neural
network optimization trajectories have been proposed to possess fractal
structure, leading to bounds and generalization measures based on notions of
fractal dimension on these trajectories. Prominently, both the Hausdorff
dimension and the persistent homology dimension have been proposed to correlate
with generalization gap, thus serving as a measure of generalization. This work
performs an extended evaluation of these topological generalization measures.
We demonstrate that fractal dimension fails to predict generalization of models
trained from poor initializations. We further identify that the $\ell^2$ norm
of the final parameter iterate, one of the simplest complexity measures in
learning theory, correlates more strongly with the generalization gap than
these notions of fractal dimension. Finally, our study reveals the intriguing
manifestation of model-wise double descent in persistent homology-based
generalization measures. This work lays the ground for a deeper investigation
of the causal relationships between fractal geometry, topological data
analysis, and neural network optimization.

深度学习中超参数化神经网络的泛化缺口、分形维度、持续同调维度、模型的双下降等方面进行了广泛的研究评估，通过观察发现，残差参数向量的 L2 范数与泛化缺口之间有更强的相关性。该研究为进一步探索分形几何、拓扑数据分析和神经网络优化之间的因果关系奠定了基础。

关于分形维数作为泛化度量的局限性

On the Limitations of Fractal Dimension as a Measure of Generalization

The phenomenon of model-wise double descent, where the test error peaks and
then reduces as the model size increases, is an interesting topic that has
attracted the attention of researchers due to the striking observed gap between
theory and practice \citep{Belkin2018ReconcilingMM}. Additionally, while double
descent has been observed in various tasks and architectures, the peak of
double descent can sometimes be noticeably absent or diminished, even without
explicit regularization, such as weight decay and early stopping. In this
paper, we investigate this intriguing phenomenon from the optimization
perspective and propose a simple optimization-based explanation for why double
descent sometimes occurs weakly or not at all. To the best of our knowledge, we
are the first to demonstrate that many disparate factors contributing to
model-wise double descent (initialization, normalization, batch size, learning
rate, optimization algorithm) are unified from the viewpoint of optimization:
model-wise double descent is observed if and only if the optimizer can find a
sufficiently low-loss minimum. These factors directly affect the condition
number of the optimization problem or the optimizer and thus affect the final
minimum found by the optimizer, reducing or increasing the height of the double
descent peak. We conduct a series of controlled experiments on random feature
models and two-layer neural networks under various optimization settings,
demonstrating this optimization-based unified view. Our results suggest the
following implication: Double descent is unlikely to be a problem for
real-world machine learning setups. Additionally, our results help explain the
gap between weak double descent peaks in practice and strong peaks observable
in carefully designed setups.

通过优化视角，我们研究了模型智能双下降现象，提出了一种简单的基于优化的解释，同时证明了多样的因素对于模型智能双下降的贡献，并且通过一系列实验展示了这种基于优化的统一观点，从而得出结论：模型智能双下降在实际的机器学习环境中很不可能出现问题。