Deep learning has been widely adopted across various fields, but there has
been little focus on evaluating the performance of deep learning pipelines.
With the increased use of large datasets and complex models, it has become
common to run the training process only once and compare the result to previous
benchmarks. However, this procedure can lead to imprecise comparisons due to
the variance in neural network evaluation metrics. The metric variance comes
from the randomness inherent in the training process of deep learning
pipelines. Traditional solutions such as running the training process multiple
times are usually not feasible in deep learning due to computational
limitations. In this paper, we propose a new metric framework, Calibrated Loss
Metric, that addresses this issue by reducing the variance in its vanilla
counterpart. As a result, the new metric has a higher accuracy to detect
effective modeling improvement. Our approach is supported by theoretical
justifications and extensive experimental validations in the context of Deep
Click-Through Rate Prediction Models.

通过减少方差提高准确性，我们提出了一种新的指标框架 —— 校准损失指标，以解决深度学习管道评估过程中的度量方差问题，并通过在深度点击率预测模型中的理论论证和大量实验验证支持我们的方法。