Over the past several years progress in designing better neural network
architectures for visual recognition has been substantial. To help sustain this
rate of progress, in this work we propose to reexamine the methodology for
comparing network architectures. In particular, we introduce a new comparison
paradigm of distribution estimates, in which network design spaces are compared
by applying statistical techniques to populations of sampled models, while
controlling for confounding factors like network complexity. Compared to
current methodologies of comparing point and curve estimates of model families,
distribution estimates paint a more complete picture of the entire design
landscape. As a case study, we examine design spaces used in neural
architecture search (NAS). We find significant statistical differences between
recent NAS design space variants that have been largely overlooked.
Furthermore, our analysis reveals that the design spaces for standard model
families like ResNeXt can be comparable to the more complex ones used in recent
NAS work. We hope these insights into distribution analysis will enable more
robust progress toward discovering better networks for visual recognition.

本研究提出了分布估计的新的比较神经网络架构方法，揭示了最新神经架构搜索设计空间变体之间的显著统计差异，并发现标准模型家族的设计空间可以与最近神经架构搜索工作中使用的更复杂的设计空间可以相媲美，以期为发现更好的视觉识别网络提供更健壮的进展。