In this work, we develop a neural architecture search algorithm, termed
Resbuilder, that develops ResNet architectures from scratch that achieve high
accuracy at moderate computational cost. It can also be used to modify existing
architectures and has the capability to remove and insert ResNet blocks, in
this way searching for suitable architectures in the space of ResNet
architectures. In our experiments on different image classification datasets,
Resbuilder achieves close to state-of-the-art performance while saving
computational cost compared to off-the-shelf ResNets. Noteworthy, we once tune
the parameters on CIFAR10 which yields a suitable default choice for all other
datasets. We demonstrate that this property generalizes even to industrial
applications by applying our method with default parameters on a proprietary
fraud detection dataset.

我们开发了一种名为 Resbuilder 的神经结构搜索算法，从头开始开发 ResNet 架构，在适度的计算成本下实现高准确性。它还可以用于修改现有的架构，并具有删除和插入 ResNet 块的功能，从而在 ResNet 架构的空间中搜索适合的架构。在不同的图像分类数据集上的实验证明，与现成的 ResNet 相比，Resbuilder 在节省计算成本的同时实现了接近最先进的性能。值得注意的是，我们在 CIFAR10 上调整参数，这为所有其他数据集提供了一个合适的默认选择。通过在一个专有的欺诈检测数据集上应用我们的方法和默认参数，我们证明了这种性质甚至适用于工业应用。

ResBuilder: 基于残差结构的深度自动学习

ResBuilder: Automated Learning of Depth with Residual Structures

Neural ordinary differential equations (NODEs) treat computation of
intermediate feature vectors as trajectories of ordinary differential equation
parameterized by a neural network. In this paper, we propose a novel model,
delay differential neural networks (DDNN), inspired by delay differential
equations (DDEs). The proposed model considers the derivative of the hidden
feature vector as a function of the current feature vector and past feature
vectors (history). The function is modelled as a neural network and
consequently, it leads to continuous depth alternatives to many recent ResNet
variants. We propose two different DDNN architectures, depending on the way
current and past feature vectors are considered. For training DDNNs, we provide
a memory-efficient adjoint method for computing gradients and back-propagate
through the network. DDNN improves the data efficiency of NODE by further
reducing the number of parameters without affecting the generalization
performance. Experiments conducted on synthetic and real-world image
classification datasets such as Cifar10 and Cifar100 show the effectiveness of
the proposed models.

本文介绍了基于延迟微分方程（DDEs）的一种新型模型延迟微分神经网络（DDNN），其包括两种不同的架构。通过提供一种内存高效的方法计算梯度并反向传播至网络中，DDNN 显著减少了一些最近 ResNet 变体的参数数量并提高了图像分类数据集，例如 Cifar10 和 Cifar100 的效果。

延迟微分神经网络

Delay Differential Neural Networks

In this paper we propose a novel decomposition method based on filter group
approximation, which can significantly reduce the redundancy of deep
convolutional neural networks (CNNs) while maintaining the majority of feature
representation. Unlike other low-rank decomposition algorithms which operate on
spatial or channel dimension of filters, our proposed method mainly focuses on
exploiting the filter group structure for each layer. For several commonly used
CNN models, including VGG and ResNet, our method can reduce over 80%
floating-point operations (FLOPs) with less accuracy drop than state-of-the-art
methods on various image classification datasets. Besides, experiments
demonstrate that our method is conducive to alleviating degeneracy of the
compressed network, which hurts the convergence and performance of the network.

该论文提出了一种基于滤波器组逼近的新型分解方法，该方法可以显著减少深度卷积神经网络中的冗余，同时保持大部分特征表示，通过在每层中利用滤波器组结构，相比于其他基于低秩分解算法，我们的方法可以在多个图像分类数据集上减少 80% 以上的算术运算量（FLOPs），并减少精度损失，另外，实验证明我们的方法有助于缓解压缩网络的退化现象，从而提高网络的收敛性和性能。