Click-Through Rate (CTR) prediction holds paramount significance in online
advertising and recommendation scenarios. Despite the proliferation of recent
CTR prediction models, the improvements in performance have remained limited,
as evidenced by open-source benchmark assessments. Current researchers tend to
focus on developing new models for various datasets and settings, often
neglecting a crucial question: What is the key challenge that truly makes CTR
prediction so demanding?
In this paper, we approach the problem of CTR prediction from an optimization
perspective. We explore the typical data characteristics and optimization
statistics of CTR prediction, revealing a strong positive correlation between
the top hessian eigenvalue and feature frequency. This correlation implies that
frequently occurring features tend to converge towards sharp local minima,
ultimately leading to suboptimal performance. Motivated by the recent
advancements in sharpness-aware minimization (SAM), which considers the
geometric aspects of the loss landscape during optimization, we present a
dedicated optimizer crafted for CTR prediction, named Helen. Helen incorporates
frequency-wise Hessian eigenvalue regularization, achieved through adaptive
perturbations based on normalized feature frequencies.
Empirical results under the open-source benchmark framework underscore
Helen's effectiveness. It successfully constrains the top eigenvalue of the
Hessian matrix and demonstrates a clear advantage over widely used optimization
algorithms when applied to seven popular models across three public benchmark
datasets on BARS. Our code locates at github.com/NUS-HPC-AI-Lab/Helen.

通过优化的角度来解决 CTR 预测问题，研究发现 CTR 预测中最大的挑战在于特征频率与最大海森矩阵特征值之间的正相关关系，即频繁出现的特征倾向于收敛于尖锐的局部最小值，从而导致亚优化性能。基于锐度感知最小化的最新进展，我们提出了一种专门为 CTR 预测设计的优化器，命名为 Helen。Helen 通过基于归一化特征频率的自适应扰动来实现特征频率相关的海森矩阵特征值正则化。在开源基准框架下的实证结果突显了 Helen 的有效性。它成功地约束了海森矩阵的最大特征值，并在三个公共基准数据集上应用于七种流行模型时，与广泛使用的优化算法相比具有明显优势。我们的代码位于 github.com/NUS-HPC-AI-Lab/Helen。

Helen: 频率导向 Hessian 特征值正则化优化 CTR 预测模型

Helen: Optimizing CTR Prediction Models with Frequency-wise Hessian  Eigenvalue Regularization

The phenomenon of model-wise double descent, where the test error peaks and
then reduces as the model size increases, is an interesting topic that has
attracted the attention of researchers due to the striking observed gap between
theory and practice \citep{Belkin2018ReconcilingMM}. Additionally, while double
descent has been observed in various tasks and architectures, the peak of
double descent can sometimes be noticeably absent or diminished, even without
explicit regularization, such as weight decay and early stopping. In this
paper, we investigate this intriguing phenomenon from the optimization
perspective and propose a simple optimization-based explanation for why double
descent sometimes occurs weakly or not at all. To the best of our knowledge, we
are the first to demonstrate that many disparate factors contributing to
model-wise double descent (initialization, normalization, batch size, learning
rate, optimization algorithm) are unified from the viewpoint of optimization:
model-wise double descent is observed if and only if the optimizer can find a
sufficiently low-loss minimum. These factors directly affect the condition
number of the optimization problem or the optimizer and thus affect the final
minimum found by the optimizer, reducing or increasing the height of the double
descent peak. We conduct a series of controlled experiments on random feature
models and two-layer neural networks under various optimization settings,
demonstrating this optimization-based unified view. Our results suggest the
following implication: Double descent is unlikely to be a problem for
real-world machine learning setups. Additionally, our results help explain the
gap between weak double descent peaks in practice and strong peaks observable
in carefully designed setups.

通过优化视角，我们研究了模型智能双下降现象，提出了一种简单的基于优化的解释，同时证明了多样的因素对于模型智能双下降的贡献，并且通过一系列实验展示了这种基于优化的统一观点，从而得出结论：模型智能双下降在实际的机器学习环境中很不可能出现问题。

优化在双下降中的作用理解

Understanding the Role of Optimization in Double Descent

Knowledge distillation is a popular approach for enhancing the performance of
``student'' models, with lower representational capacity, by taking advantage
of more powerful ``teacher'' models. Despite its apparent simplicity and
widespread use, the underlying mechanics behind knowledge distillation (KD) are
still not fully understood. In this work, we shed new light on the inner
workings of this method, by examining it from an optimization perspective. We
show that, in the context of linear and deep linear models, KD can be
interpreted as a novel type of stochastic variance reduction mechanism. We
provide a detailed convergence analysis of the resulting dynamics, which hold
under standard assumptions for both strongly-convex and non-convex losses,
showing that KD acts as a form of \emph{partial variance reduction}, which can
reduce the stochastic gradient noise, but may not eliminate it completely,
depending on the properties of the ``teacher'' model. Our analysis puts further
emphasis on the need for careful parametrization of KD, in particular w.r.t.
the weighting of the distillation loss, and is validated empirically on both
linear models and deep neural networks.

本文揭示了知识蒸馏方法的内在机制，对应于一种新型的随机方差缩减机制。通过在线性模型和深度线性模型的背景下实现这一方法，表明知识蒸馏可以降低随机梯度噪声，从而减少模型复杂度，但是需要进行参数化的调整。