Since deep neural networks were developed, they have made huge contributions
to everyday lives. Machine learning provides more rational advice than humans
are capable of in almost every aspect of daily life. However, despite this
achievement, the design and training of neural networks are still challenging
and unpredictable procedures. To lower the technical thresholds for common
users, automated hyper-parameter optimization (HPO) has become a popular topic
in both academic and industrial areas. This paper provides a review of the most
essential topics on HPO. The first section introduces the key hyper-parameters
related to model training and structure, and discusses their importance and
methods to define the value range. Then, the research focuses on major
optimization algorithms and their applicability, covering their efficiency and
accuracy especially for deep learning networks. This study next reviews major
services and toolkits for HPO, comparing their support for state-of-the-art
searching algorithms, feasibility with major deep learning frameworks, and
extensibility for new modules designed by users. The paper concludes with
problems that exist when HPO is applied to deep learning, a comparison between
optimization algorithms, and prominent approaches for model evaluation with
limited computational resources.

利用自动超参数优化来设计和训练深度学习中的神经网络仍然是一个令人困惑和具挑战性的过程。本文对多种重要的自动超参数优化相关主题进行了回顾和研究：关键超参数、主要优化算法、超参数搜索框架、模型性能评估等。