Stochastic optimization algorithms have been successfully applied in several
domains to find optimal solutions. Because of the ever-growing complexity of
the integrated systems, novel stochastic algorithms are being proposed, which
makes the task of the performance analysis of the algorithms extremely
important. In this paper, we provide a novel ranking scheme to rank the
algorithms over multiple single-objective optimization problems. The results of
the algorithms are compared using a robust bootstrapping-based hypothesis
testing procedure that is based on the principles of severity. Analogous to the
football league scoring scheme, we propose pairwise comparison of algorithms as
in league competition. Each algorithm accumulates points and a performance
metric of how good or bad it performed against other algorithms analogous to
goal differences metric in football league scoring system. The goal differences
performance metric can not only be used as a tie-breaker but also be used to
obtain a quantitative performance of each algorithm. The key novelty of the
proposed ranking scheme is that it takes into account the performance of each
algorithm considering the magnitude of the achieved performance improvement
along with its practical relevance and does not have any distributional
assumptions. The proposed ranking scheme is compared to classical hypothesis
testing and the analysis of the results shows that the results are comparable
and our proposed ranking showcases many additional benefits.

提出了一种新的排名方案，利用基于鲁棒的自助法假设检验程序对多个单目标优化问题的算法进行排名，并考虑了算法的性能改进的大小和实际相关性，与传统假设检验相比，提出的排名方案具有可比性和许多额外的好处。