The performance of fine-tuning pre-trained language models largely depends on
the hyperparameter configuration. In this paper, we investigate the performance
of modern hyperparameter optimization methods (HPO) on fine-tuning pre-trained
language models. First, we study and report three HPO algorithms' performances
on fine-tuning two state-of-the-art language models on the GLUE dataset. We
find that using the same time budget, HPO often fails to outperform grid search
due to two reasons: insufficient time budget and overfitting. We propose two
general strategies and an experimental procedure to systematically troubleshoot
HPO's failure cases. By applying the procedure, we observe that HPO can succeed
with more appropriate settings in the search space and time budget; however, in
certain cases overfitting remains. Finally, we make suggestions for future
work. Our implementation can be found in
this https URL

本文研究了模型微调和超参数优化的关系，并提出了两种一般策略以及一个实验流程，以系统地解决超参数优化在微调预训练语言模型时的失败案例。通过实验，我们发现在更适当的搜索空间和时间预算设置下，超参数优化可以取得成功，但在某些情况下，过拟合仍然存在。