Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem -- loss aggregation and label aggregation -- by characterizing their Bayes-optimal solutions. Based on this, we show that while both methods can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.

本研究解决了如何将多个二元目标标签综合为一个一致的排名问题，尤其是在不同人类注释者提供标签的情况下。论文形式分析了损失聚合和标签聚合两种方法，指出虽然两者均可产生帕累托最优解，但损失聚合可能导致标签独裁现象，因此标签聚合通常更为可取。研究结果为这一领域的实际应用提供了重要的指导。

多标签下的二分排名：损失与标签聚合的比较