In the rapidly evolving domain of Recommender Systems (RecSys), new algorithms frequently claim state-of-the-art performance based on evaluations over a limited set of arbitrarily selected datasets. However, this approach may fail to holistically reflect their effectiveness due to the significant impact of dataset characteristics on algorithm performance. Addressing this deficiency, this paper introduces a novel benchmarking methodology to facilitate a fair and robust comparison of RecSys algorithms, thereby advancing evaluation practices. By utilizing a diverse set of $30$ open datasets, including two introduced in this work, and evaluating $11$ collaborative filtering algorithms across $9$ metrics, we critically examine the influence of dataset characteristics on algorithm performance. We further investigate the feasibility of aggregating outcomes from multiple datasets into a unified ranking. Through rigorous experimental analysis, we validate the reliability of our methodology under the variability of datasets, offering a benchmarking strategy that balances quality and computational demands. This methodology enables a fair yet effective means of evaluating RecSys algorithms, providing valuable guidance for future research endeavors.

通过使用多样化的30个开放数据集和11个协同过滤算法在9个评估指标下的评估，本论文引入了一种新的基准方法，以促进推荐系统算法的公正和稳健比较，从而推进评估实践。通过严格的实验分析，验证了在数据集的变化性下的方法可靠性，提供了一个既公正又有效的推荐系统算法评估方法，为未来研究工作提供了有价值的指导。

从变化到稳定：推荐系统基准实践的进一步发展