In this paper, we tackle the problem of convolutional neural network design.
Instead of focusing on the design of the overall architecture, we investigate a
design space that is usually overlooked, i.e. adjusting the channel
configurations of predefined networks. We find that this adjustment can be
achieved by shrinking widened baseline networks and leads to superior
performance. Based on that, we articulate the heterogeneity hypothesis: with
the same training protocol, there exists a layer-wise differentiated network
architecture (LW-DNA) that can outperform the original network with regular
channel configurations but with a lower level of model complexity.
The LW-DNA models are identified without extra computational cost or training
time compared with the original network. This constraint leads to controlled
experiments which direct the focus to the importance of layer-wise specific
channel configurations. LW-DNA models come with advantages related to
overfitting, i.e. the relative relationship between model complexity and
dataset size. Experiments are conducted on various networks and datasets for
image classification, visual tracking and image restoration. The resultant
LW-DNA models consistently outperform the baseline models. Code is available at
this https URL

本文探讨了卷积神经网络设计的问题。我们研究了一个经常被忽视的设计空间，即调整预定义网络的通道配置。我们发现这种调整可以通过缩小加宽的基础网络来实现，并且具有优越的性能。基于此，我们提出了异质性假设：采用同样的训练协议，存在一种逐层差异的网络架构（LW-DNA），可以在低模型复杂度的情况下优于具有常规通道配置的原始网络。LW-DNA 模型可以在不增加额外计算成本或训练时间的情况下被识别出来。实验表明，LW-DNA 模型在各种网络和数据集上都表现出色。

异构假设：寻找逐层分化的网络架构

The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network  Architectures

We study how to set channel numbers in a neural network to achieve better
accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or
model size). A simple and one-shot solution, named AutoSlim, is presented.
Instead of training many network samples and searching with reinforcement
learning, we train a single slimmable network to approximate the network
accuracy of different channel configurations. We then iteratively evaluate the
trained slimmable model and greedily slim the layer with minimal accuracy drop.
By this single pass, we can obtain the optimized channel configurations under
different resource constraints. We present experiments with MobileNet v1,
MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We
show significant improvements over their default channel configurations. We
also achieve better accuracy than recent channel pruning methods and neural
architecture search methods.
Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at
305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2
(301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our
AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3%
better accuracy than MobileNet-v1 (569M FLOPs). Code and models will be
available at: this https URL

该研究提出了一种名为 AutoSlim 的单次解决方案，通过单轮迭代评估训练后的可裁剪模型并贪心地裁剪使准确度下降最小的层来获得不同资源限制下的优化通道配置，从而在 ImageNet 分类等任务中取得显著的性能提升。