We run an independent comparison of all hyperparameter optimization (hyperopt) engines available in the Ray Tune library. We introduce two ways to normalize and aggregate statistics across data sets and models, one rank-based, and another one sandwiching the score between the random search score and the full grid search score. This affords us i) to rank the hyperopt engines, ii) to make generalized and statistically significant statements on how much they improve over random search, and iii) to make recommendations on which engine should be used to hyperopt a given learning algorithm. We find that most engines beat random search, but that only three of them (HEBO, AX, and BlendSearch) clearly stand out. We also found that some engines seem to specialize in hyperopting certain learning algorithms, which makes it tricky to use hyperopt in comparison studies, since the choice of the hyperopt technique may favor some of the models in the comparison.

我们在Ray Tune库中对所有可用的超参数优化（hyperopt）引擎进行了独立比较。我们引入了两种方法来归一化和聚合数据集和模型之间的统计数据，一种是基于排名的方法，另一种是将分数夹在随机搜索分数和完整网格搜索分数之间。这使得我们能够：i）对超参数优化引擎进行排名，ii）对其相对于随机搜索的改进程度进行一般化和统计学上显著的说明，iii）对应选择的引擎在超参优化特定学习算法上进行推荐。我们发现大多数引擎都能击败随机搜索，但只有三个引擎（HEBO、AX和BlendSearch）明显突出。我们还发现一些引擎似乎专注于超参优化特定的学习算法，这使得在比较研究中使用超参数优化技术变得棘手，因为超参数优化技术的选择可能对比较中的某些模型有利。

在表格数据上对超参数优化引擎进行系统性研究