In many learning situations, resources at inference time are significantly
more constrained than resources at training time. This paper studies a general
paradigm, called Differentiable ARchitecture Compression (DARC), that combines
model compression and architecture search to learn models that are
resource-efficient at inference time. Given a resource-intensive base
architecture, DARC utilizes the training data to learn which sub-components can
be replaced by cheaper alternatives. The high-level technique can be applied to
any neural architecture, and we report experiments on state-of-the-art
convolutional neural networks for image classification. For a WideResNet with
$97.2\%$ accuracy on CIFAR-10, we improve single-sample inference speed by
$2.28\times$ and memory footprint by $5.64\times$, with no accuracy loss. For a
ResNet with $79.15\%$ Top1 accuracy on ImageNet, we improve batch inference
speed by $1.29\times$ and memory footprint by $3.57\times$ with $1\%$ accuracy
loss. We also give theoretical Rademacher complexity bounds in simplified
cases, showing how DARC avoids overfitting despite over-parameterization.

本文提出一种新的范例叫做 Differentiable ARchitecture Compression (DARC)，这种方法结合了模型压缩和架构搜索的优势从而能够学习出在推理时更加高效的模型，DARC 能够应用于任何神经架构，并在现代卷积神经网络进行图像分类的实验结果表明，对于 CIFAR-10 上的 WideResNet，我们将单样本推理速度提高了 2.28 倍，内存占用降低了 5.64 倍，而准确率没有损失。对于在 ImageNet 数据集上具有 79.15% Top1 准确率的 ResNet，我们提高了 1.29 倍的批量推理速度和 3.57 倍的内存占用，并且仅损失了 1% 的准确率。此外，我们提供了简化的理论 Rademacher 复杂度界限，展示了 DARC 如何在数据过度载入的情况下避免了过度拟合的问题。