Neural Architecture Search (NAS) yields state-of-the-art neural networks that
outperform their best manually-designed counterparts. However, previous NAS
methods search for architectures under one set of training hyper-parameters
(i.e., a training recipe), overlooking superior architecture-recipe
combinations. To address this, we present Neural Architecture-Recipe Search
(NARS) to search both (a) architectures and (b) their corresponding training
recipes, simultaneously. NARS utilizes an accuracy predictor that scores
architecture and training recipes jointly, guiding both sample selection and
ranking. Furthermore, to compensate for the enlarged search space, we leverage
"free" architecture statistics (e.g., FLOP count) to pretrain the predictor,
significantly improving its sample efficiency and prediction reliability. After
training the predictor via constrained iterative optimization, we run fast
evolutionary searches in just CPU minutes to generate architecture-recipe pairs
for a variety of resource constraints, called FBNetV3. FBNetV3 makes up a
family of state-of-the-art compact neural networks that outperform both
automatically and manually-designed competitors. For example, FBNetV3 matches
both EfficientNet and ResNeSt accuracy on ImageNet with up to 2.0x and 7.1x
fewer FLOPs, respectively. Furthermore, FBNetV3 yields significant performance
gains for downstream object detection tasks, improving mAP despite 18% fewer
FLOPs and 34% fewer parameters than EfficientNet-based equivalents.

提出了一种同时搜索神经架构和相应的训练参数的新方法 —— 神经架构及训练参数搜索（NARS），其中利用了免费的架构统计信息（如 FLOP 计数）来预先训练准确性预测器，使用受约束的迭代优化方法训练预测器，然后快速进行进化搜索来生成一系列精简的神经网络，称为 FBNetV3，它在资源限制条件下的准确率优于其他自动或手动设计的对手。