In the rapidly evolving domain of Recommender Systems (RecSys), new
algorithms frequently claim state-of-the-art performance based on evaluations
over a limited set of arbitrarily selected datasets. However, this approach may
fail to holistically reflect their effectiveness due to the significant impact
of dataset characteristics on algorithm performance. Addressing this
deficiency, this paper introduces a novel benchmarking methodology to
facilitate a fair and robust comparison of RecSys algorithms, thereby advancing
evaluation practices. By utilizing a diverse set of $30$ open datasets,
including two introduced in this work, and evaluating $11$ collaborative
filtering algorithms across $9$ metrics, we critically examine the influence of
dataset characteristics on algorithm performance. We further investigate the
feasibility of aggregating outcomes from multiple datasets into a unified
ranking. Through rigorous experimental analysis, we validate the reliability of
our methodology under the variability of datasets, offering a benchmarking
strategy that balances quality and computational demands. This methodology
enables a fair yet effective means of evaluating RecSys algorithms, providing
valuable guidance for future research endeavors.

通过使用多样化的 30 个开放数据集和 11 个协同过滤算法在 9 个评估指标下的评估，本论文引入了一种新的基准方法，以促进推荐系统算法的公正和稳健比较，从而推进评估实践。通过严格的实验分析，验证了在数据集的变化性下的方法可靠性，提供了一个既公正又有效的推荐系统算法评估方法，为未来研究工作提供了有价值的指导。

从变化到稳定：推荐系统基准实践的进一步发展

From Variability to Stability: Advancing RecSys Benchmarking Practices

Collaborative filtering algorithms have the advantage of not requiring
sensitive user or item information to provide recommendations. However, they
still suffer from fairness related issues, like popularity bias. In this work,
we argue that popularity bias often leads to other biases that are not obvious
when additional user or item information is not provided to the researcher. We
examine our hypothesis in the book recommendation case on a commonly used
dataset with book ratings. We enrich it with author information using publicly
available external sources. We find that popular books are mainly written by US
citizens in the dataset, and that these books tend to be recommended
disproportionally by popular collaborative filtering algorithms compared to the
users' profiles. We conclude that the societal implications of popularity bias
should be further examined by the scholar community.

本文研究合作过滤算法中的公平性问题，特别是流行度偏见对推荐结果的影响，并利用外部来源的作者信息富化图书评分数据集。结果表明，热门的合作过滤算法存在天空偏见，推荐结果偏向于美国籍作者编写的流行书籍。本研究认为应该进一步研究流行度偏见的社会影响。