Training machine learning models inherently involves a resource-intensive and
noisy iterative learning procedure that allows epoch-wise monitoring of the
model performance. However, in multi-objective hyperparameter optimization
scenarios, the insights gained from the iterative learning procedure typically
remain underutilized. We notice that tracking the model performance across
multiple epochs under a hyperparameter setting creates a trajectory in the
objective space and that trade-offs along the trajectories are often overlooked
despite their potential to offer valuable insights to decision-making for model
retraining. Therefore, in this study, we propose to enhance the multi-objective
hyperparameter optimization problem by having training epochs as an additional
decision variable to incorporate trajectory information. Correspondingly, we
present a novel trajectory-based multi-objective Bayesian optimization
algorithm characterized by two features: 1) an acquisition function that
captures the improvement made by the predictive trajectory of any
hyperparameter setting and 2) a multi-objective early stopping mechanism that
determines when to terminate the trajectory to maximize epoch efficiency.
Numerical experiments on diverse synthetic simulations and hyperparameter
tuning benchmarks indicate that our algorithm outperforms the state-of-the-art
multi-objective optimizers in both locating better trade-offs and tuning
efficiency.

本研究提出了一种基于轨迹的多目标贝叶斯优化算法，通过考虑训练轮次作为附加决策变量，充分利用迭代学习过程中的轨迹信息来提升多目标超参数优化问题的性能。实验证明，该算法在寻找更好的权衡和提高调优效率方面优于现有的多目标优化器。