The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse
models, possibly without calibration. An important example of AUC application
is the evaluation and benchmarking of models that predict faithfulness of
generated text. But we show that the AUC yields an academic and optimistic
notion of accuracy that can misalign with the actual accuracy observed in
application, yielding significant changes in benchmark rankings. To paint a
more realistic picture of downstream model performance (and prepare a model for
actual application), we explore different calibration modes, testing
calibration data and method.

使用 AUC 度量方式评估和比较不同模型，但该方法在应用中可能与实际精度偏差较大，因此需探索不同的校准模式和测试校准数据与方法来获取更真实的模型性能。

薛定谔的门槛：当 AUC 不能预测准确度

Schroedinger's Threshold: When the AUC doesn't predict Accuracy

Federated learning has attracted increasing attention due to the promise of
balancing privacy and large-scale learning; numerous approaches have been
proposed. However, most existing approaches focus on problems with balanced
data, and prediction performance is far from satisfactory for many real-world
applications where the number of samples in different classes is highly
imbalanced. To address this challenging problem, we developed a novel federated
learning method for imbalanced data by directly optimizing the area under curve
(AUC) score. In particular, we formulate the AUC maximization problem as a
federated compositional minimax optimization problem, develop a local
stochastic compositional gradient descent ascent with momentum algorithm, and
provide bounds on the computational and communication complexities of our
algorithm. To the best of our knowledge, this is the first work to achieve such
favorable theoretical results. Finally, extensive experimental results confirm
the efficacy of our method.

本研究提出了一种针对非平衡数据的联邦学习方法，通过直接优化曲线下面积（AUC）分数，采用分解最小最大优化问题和随机梯度下降优化算法，以及提供算法的计算和通信复杂性界限，达到了有效性的实验结果。