Differential Neural Architecture Search (NAS) requires all layer choices to
be held in memory simultaneously; this limits the size of both search space and
final architecture. In contrast, Probabilistic NAS, such as PARSEC, learns a
distribution over high-performing architectures, and uses only as much memory
as needed to train a single model. Nevertheless, it needs to sample many
architectures, making it computationally expensive for searching in an
extensive space. To solve these problems, we propose a sampling method adaptive
to the distribution entropy, drawing more samples to encourage explorations at
the beginning, and reducing samples as learning proceeds. Furthermore, to
search fast in the multi-variate space, we propose a coarse-to-fine strategy by
using a factorized distribution at the beginning which can reduce the number of
architecture parameters by over an order of magnitude. We call this method Fast
Probabilistic NAS (FP-NAS). Compared with PARSEC, it can sample 64% fewer
architectures and search 2.1x faster. Compared with FBNetV2, FP-NAS is 1.9x -
3.5x faster, and the searched models outperform FBNetV2 models on ImageNet.
FP-NAS allows us to expand the giant FBNetV2 space to be wider (i.e. larger
channel choices) and deeper (i.e. more blocks), while adding Split-Attention
block and enabling the search over the number of splits. When searching a model
of size 0.4G FLOPS, FP-NAS is 132x faster than EfficientNet, and the searched
FP-NAS-L0 model outperforms EfficientNet-B0 by 0.7% accuracy. Without using any
architecture surrogate or scaling tricks, we directly search large models up to
1.0G FLOPS. Our FP-NAS-L2 model with simple distillation outperforms BigNAS-XL
with advanced in-place distillation by 0.7% accuracy using similar FLOPS.

提出了一种自适应于分布熵的采样方法，从而实现快速的多元空间神经体系结构搜索，成为 FP-NAS。通过 FP-NAS，可以使搜索空间变得更大、更深、更准确，速度比其他方法快 2.1 倍 - 3.5 倍，并且可以直接搜索高达 1.0G FLOPS 的大模型，在数字图像分类方面表现出色。