Despite the recent success in a plethora of hyperparameter optimization (HPO)
methods for machine learning (ML) models, the intricate interplay between model
hyperparameters (HPs) and predictive losses (a.k.a fitness), which is a key
prerequisite for understanding HPO, remain notably underexplored in our
community. This results in limited explainability in the HPO process, rendering
a lack of human trust and difficulties in pinpointing algorithm bottlenecks. In
this paper, we aim to shed light on this black box by conducting large-scale
fitness landscape analysis (FLA) on 1,500 HP loss landscapes of 6 ML models
with more than 11 model configurations, across 67 datasets and different levels
of fidelities. We reveal the first unified, comprehensive portrait of their
topographies in terms of smoothness, neutrality and modality. We also show that
such properties are highly transferable across datasets and fidelities,
providing fundamental evidence for the success of multi-fidelity and transfer
learning methods. These findings are made possible by developing a dedicated
FLA framework that incorporates a combination of visual and quantitative
measures. We further demonstrate the potential of this framework by analyzing
the NAS-Bench-101 landscape, and we believe it is able to faciliate fundamental
understanding of a broader range of AutoML tasks.

通过进行大规模的 1,500 个超参数与损失地形分析，研究揭示了机器学习模型中超参数和预测损失之间相互作用的关键要素，对多精度和迁移学习方法的成功提供了基础证据，并开发了专门的分析框架来促进对更广泛的 AutoML 任务的基本理解。