We conduct a comprehensive investigation into the dynamics of gradient
descent using large-order constant step-sizes in the context of quadratic
regression models. Within this framework, we reveal that the dynamics can be
encapsulated by a specific cubic map, naturally parameterized by the step-size.
Through a fine-grained bifurcation analysis concerning the step-size parameter,
we delineate five distinct training phases: (1) monotonic, (2) catapult, (3)
periodic, (4) chaotic, and (5) divergent, precisely demarcating the boundaries
of each phase. As illustrations, we provide examples involving phase retrieval
and two-layer neural networks employing quadratic activation functions and
constant outer-layers, utilizing orthogonal training data. Our simulations
indicate that these five phases also manifest with generic non-orthogonal data.
We also empirically investigate the generalization performance when training in
the various non-monotonic (and non-divergent) phases. In particular, we observe
that performing an ergodic trajectory averaging stabilizes the test error in
non-monotonic (and non-divergent) phases.

通过对大步长梯度下降在二次回归模型中的动力学进行全面调查，揭示了动力学可以由特定的三次映射来描述，并通过细致的分叉分析划分了五个不同的训练阶段，同时研究了非单调和非发散阶段的泛化性能。