Machine learning has achieved remarkable success over the past couple of
decades, often attributed to a combination of algorithmic innovations and the
availability of high-quality data available at scale. However, a third critical
component is the fine-tuning of hyperparameters, which plays a pivotal role in
achieving optimal model performance. Despite its significance, hyperparameter
optimization (HPO) remains a challenging task for several reasons. Many HPO
techniques rely on naive search methods or assume that the loss function is
smooth and continuous, which may not always be the case. Traditional methods,
like grid search and Bayesian optimization, often struggle to quickly adapt and
efficiently search the loss landscape. Grid search is computationally
expensive, while Bayesian optimization can be slow to prime. Since the search
space for HPO is frequently high-dimensional and non-convex, it is often
challenging to efficiently find a global minimum. Moreover, optimal
hyperparameters can be sensitive to the specific dataset or task, further
complicating the search process. To address these issues, we propose a new
hyperparameter optimization method, HomOpt, using a data-driven approach based
on a generalized additive model (GAM) surrogate combined with homotopy
optimization. This strategy augments established optimization methodologies to
boost the performance and effectiveness of any given method with faster
convergence to the optimum on continuous, discrete, and categorical domain
spaces. We compare the effectiveness of HomOpt applied to multiple optimization
techniques (e.g., Random Search, TPE, Bayes, and SMAC) showing improved
objective performance on many standardized machine learning benchmarks and
challenging open-set recognition tasks.

使用基于广义加法模型（GAM）替代与同伦优化结合的数据驱动方法，我们提出了一种新的超参数优化方法 HomOpt，以提高现有方法在连续、离散和分类域空间上的性能和效果，并在多个优化技术（如随机搜索、TPE、贝叶斯和 SMAC）应用 HomOpt，展示了对许多标准机器学习基准和具有挑战性的开集识别任务的改进性能。