Batch Normalization is a key component in almost all state-of-the-art image
classifiers, but it also introduces practical challenges: it breaks the
independence between training examples within a batch, can incur compute and
memory overhead, and often results in unexpected bugs. Building on recent
theoretical analyses of deep ResNets at initialization, we propose a simple set
of analysis tools to characterize signal propagation on the forward pass, and
leverage these tools to design highly performant ResNets without activation
normalization layers. Crucial to our success is an adapted version of the
recently proposed Weight Standardization. Our analysis tools show how this
technique preserves the signal in networks with ReLU or Swish activation
functions by ensuring that the per-channel activation means do not grow with
depth. Across a range of FLOP budgets, our networks attain performance
competitive with the state-of-the-art EfficientNets on ImageNet.

该研究介绍了一种能够设计高效 ResNets 网络的分析工具，通过采用一种改进后的 Weight Standardization 技术，该分析工具的作用在于确保具有 ReLU 或 Swish 激活函数的信号在网络的深度方向上不发生偏移以达到在 FLOP 预算范围内与当今最先进的 EfficientNets 在 ImageNet 上的性能竞争可比性。

特征化信号传播以缩小未标准化 ResNets 性能差距

Characterizing signal propagation to close the performance gap in  unnormalized ResNets

To obtain excellent deep neural architectures, a series of techniques are
carefully designed in EfficientNets. The giant formula for simultaneously
enlarging the resolution, depth and width provides us a Rubik's cube for neural
networks. So that we can find networks with high efficiency and excellent
performance by twisting the three dimensions. This paper aims to explore the
twisting rules for obtaining deep neural networks with minimum model sizes and
computational costs. Different from the network enlarging, we observe that
resolution and depth are more important than width for tiny networks.
Therefore, the original method, i.e., the compound scaling in EfficientNet is
no longer suitable. To this end, we summarize a tiny formula for downsizing
neural architectures through a series of smaller models derived from the
EfficientNet-B0 with the FLOPs constraint. Experimental results on the ImageNet
benchmark illustrate that our TinyNet performs much better than the smaller
version of EfficientNets using the inversed giant formula. For instance, our
TinyNet-E achieves a 59.9% Top-1 accuracy with only 24M FLOPs, which is about
1.9% higher than that of the previous best MobileNetV3 with similar
computational cost. Code will be available at
this https URL, and
this https URL

本文通过设计一系列技术，探究获得带有最小模型大小和计算成本的深度神经网络的扭曲规则。TinyNet 通过一系列 EfficientNet-B0 的小型模型总结了一种缩小神经构架的微小公式，并且实验证明 TinyNet-E 的 Top-1 准确率可以高达 59.9％，这比使用反向 Giant 公式的 EfficientNets 更高 1.9％，计算成本相似。

模型魔方：针对 TinyNets 的旋转分辨率、深度和宽度调整

Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed
resource budget, and then scaled up for better accuracy if more resources are
available. In this paper, we systematically study model scaling and identify
that carefully balancing network depth, width, and resolution can lead to
better performance. Based on this observation, we propose a new scaling method
that uniformly scales all dimensions of depth/width/resolution using a simple
yet highly effective compound coefficient. We demonstrate the effectiveness of
this method on scaling up MobileNets and ResNet.
To go even further, we use neural architecture search to design a new
baseline network and scale it up to obtain a family of models, called
EfficientNets, which achieve much better accuracy and efficiency than previous
ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3%
top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on
inference than the best existing ConvNet. Our EfficientNets also transfer well
and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%),
and 3 other transfer learning datasets, with an order of magnitude fewer
parameters. Source code is at
this https URL

本文系统研究了模型的缩放并提出了一种基于网络深度、宽度和分辨率的平衡方法，进而使用神经架构搜索设计了一种新的基线网络并扩展成一系列模型，称为 EfficientNets，这些 EfficientNets 在多项测试中取得了更好的准确性和效率，包括在 ImageNet 数据集上达到了 84.3% 的 top-1 准确率，在成为当前最佳的 ConvNet 的推理速度比其快 6.1 倍、参数更少的情况下，这是一个领先的结果。