Automated benchmarking environments aim to support researchers in
understanding how different algorithms perform on different types of
optimization problems. Such comparisons provide insights into the strengths and
weaknesses of different approaches, which can be leveraged into designing new
algorithms and into the automation of algorithm selection and configuration.
With the ultimate goal to create a meaningful benchmark set for iterative
optimization heuristics, we have recently released IOHprofiler, a software
built to create detailed performance comparisons between iterative optimization
heuristics.
With this present work we demonstrate that IOHprofiler provides a suitable
environment for automated benchmarking. We compile and assess a selection of 23
discrete optimization problems that subscribe to different types of fitness
landscapes. For each selected problem we compare performances of twelve
different heuristics, which are as of now available as baseline algorithms in
IOHprofiler.
We also provide a new module for IOHprofiler which extents the fixed-target
and fixed-budget results for the individual problems by ECDF results, which
allows one to derive aggregated performance statistics for groups of problems.

本文介绍了 IOHprofiler，一个自动化算法评估工具，以及在不同优化问题上的实践，通过比较并分析不同启发式算法的表现，提供了对各种方法优势和劣势的认识，并能进行自动算法选择和配置，为迭代优化启发式算法提供了常用的参考和标准。

使用 IOHprofiler 进行离散优化启发式算法基准测试

Benchmarking Discrete Optimization Heuristics with IOHprofiler

IOHprofiler is a new tool for analyzing and comparing iterative optimization
heuristics. Given as input algorithms and problems written in C or Python, it
provides as output a statistical evaluation of the algorithms' performance by
means of the distribution on the fixed-target running time and the fixed-budget
function values. In addition, IOHprofiler also allows to track the evolution of
algorithm parameters, making our tool particularly useful for the analysis,
comparison, and design of (self-)adaptive algorithms.
IOHprofiler is a ready-to-use software. It consists of two parts: an
experimental part, which generates the running time data, and a post-processing
part, which produces the summarizing comparisons and statistical evaluations.
The experimental part is build on the COCO software, which has been adjusted to
cope with optimization problems that are formulated as functions
$f:\mathcal{S}^n \to \R$ with $\mathcal{S}$ being a discrete alphabet of
integers. The post-processing part is our own work. It can be used as a
stand-alone tool for the evaluation of running time data of arbitrary benchmark
problems. It accepts as input files not only the output files of IOHprofiler,
but also original COCO data files. The post-processing tool is designed for an
interactive evaluation, allowing the user to chose the ranges and the precision
of the displayed data according to his/her needs.
IOHprofiler is available on GitHub at https://github.com/IOHprofiler.

IOHprofiler 是一个用于分析和比较迭代优化启发式算法的工具，可以评估算法性能并跟踪算法参数的演变，是评估任意基准问题运行时间数据的独立工具，可在 GitHub 上获取。