The phenomenon of model-wise double descent, where the test error peaks and
then reduces as the model size increases, is an interesting topic that has
attracted the attention of researchers due to the striking observed gap between
theory and practice \citep{Belkin2018ReconcilingMM}. Additionally, while double
descent has been observed in various tasks and architectures, the peak of
double descent can sometimes be noticeably absent or diminished, even without
explicit regularization, such as weight decay and early stopping. In this
paper, we investigate this intriguing phenomenon from the optimization
perspective and propose a simple optimization-based explanation for why double
descent sometimes occurs weakly or not at all. To the best of our knowledge, we
are the first to demonstrate that many disparate factors contributing to
model-wise double descent (initialization, normalization, batch size, learning
rate, optimization algorithm) are unified from the viewpoint of optimization:
model-wise double descent is observed if and only if the optimizer can find a
sufficiently low-loss minimum. These factors directly affect the condition
number of the optimization problem or the optimizer and thus affect the final
minimum found by the optimizer, reducing or increasing the height of the double
descent peak. We conduct a series of controlled experiments on random feature
models and two-layer neural networks under various optimization settings,
demonstrating this optimization-based unified view. Our results suggest the
following implication: Double descent is unlikely to be a problem for
real-world machine learning setups. Additionally, our results help explain the
gap between weak double descent peaks in practice and strong peaks observable
in carefully designed setups.

通过优化视角，我们研究了模型智能双下降现象，提出了一种简单的基于优化的解释，同时证明了多样的因素对于模型智能双下降的贡献，并且通过一系列实验展示了这种基于优化的统一观点，从而得出结论：模型智能双下降在实际的机器学习环境中很不可能出现问题。

优化在双下降中的作用理解

Understanding the Role of Optimization in Double Descent

We present Recurrent Fitting (ReFit), a neural network architecture for
single-image, parametric 3D human reconstruction. ReFit learns a
feedback-update loop that mirrors the strategy of solving an inverse problem
through optimization. At each iterative step, it reprojects keypoints from the
human model to feature maps to query feedback, and uses a recurrent-based
updater to adjust the model to fit the image better. Because ReFit encodes
strong knowledge of the inverse problem, it is faster to train than previous
regression models. At the same time, ReFit improves state-of-the-art
performance on standard benchmarks. Moreover, ReFit applies to other
optimization settings, such as multi-view fitting and single-view shape
fitting. Project website: this https URL

ReFit 是一种用于单影像参数化 3D 人体重建的神经网络架构，通过优化的方式学习反馈更新循环，其在每个迭代步骤中使用循环式更新器调整模型以更好地适应图像，提高了标准基准测试的最新性能，并适用于其他优化设置，如多视图拟合和单视图形状拟合。