Early methods in the rapidly developing field of neural architecture search
(NAS) required fully training thousands of neural networks. To reduce this
extreme computational cost, dozens of techniques have since been proposed to
predict the final performance of neural architectures. Despite the success of
such performance prediction methods, it is not well-understood how different
families of techniques compare to one another, due to the lack of an
agreed-upon evaluation metric and optimization for different constraints on the
initialization time and query time. In this work, we give the first large-scale
study of performance predictors by analyzing 31 techniques ranging from
learning curve extrapolation, to weight-sharing, to supervised learning, to
"zero-cost" proxies. We test a number of correlation- and rank-based
performance measures in a variety of settings, as well as the ability of each
technique to speed up predictor-based NAS frameworks. Our results act as
recommendations for the best predictors to use in different settings, and we
show that certain families of predictors can be combined to achieve even better
predictive power, opening up promising research directions. Our code, featuring
a library of 31 performance predictors, is available at
this https URL

本研究评估了 31 种不同的神经架构搜索性能预测器，测试了多种性能衡量方法以及它们在加速基于预测的 NAS 框架方面的有效性，结果表明有些预测器组合可以实现更好的预测能力，并提供了不同设置下最佳预测器的建议。