We prove new upper and lower bounds on the VC-dimension of deep neural
networks with the ReLU activation function. These bounds are tight for almost
the entire range of parameters. Letting $W$ be the number of weights and $L$ be
the number of layers, we prove that the VC-dimension is $O(W L \log(W))$, and
provide examples with VC-dimension $\Omega( W L \log(W/L) )$. This improves
both the previously known upper bounds and lower bounds. In terms of the number
$U$ of non-linear units, we prove a tight bound $\Theta(W U)$ on the
VC-dimension. All of these bounds generalize to arbitrary piecewise linear
activation functions, and also hold for the pseudodimensions of these function
classes.
Combined with previous results, this gives an intriguing range of
dependencies of the VC-dimension on depth for networks with different
non-linearities: there is no dependence for piecewise-constant, linear
dependence for piecewise-linear, and no more than quadratic dependence for
general piecewise-polynomial.

本文证明了深度神经网络中使用 ReLU 激活函数的 VC 维度上下界，并提供了具有不同 VC 维度的示例。此外，本文还证明了 VC 维度与权重和非线性单元数量之间存在紧密的关系，这些结论适用于任意分段线性激活函数。

分段线性神经网络的近乎紧凑 VC 维度和伪维度限制

Nearly-tight VC-dimension and pseudodimension bounds for piecewise  linear neural networks

Recently, convolutional neural networks (CNNs) have been used as a powerful
tool to solve many problems of machine learning and computer vision. In this
paper, we aim to provide insight on the property of convolutional neural
networks, as well as a generic method to improve the performance of many CNN
architectures. Specifically, we first examine existing CNN models and observe
an intriguing property that the filters in the lower layers form pairs (i.e.,
filters with opposite phase). Inspired by our observation, we propose a novel,
simple yet effective activation scheme called concatenated ReLU (CRelu) and
theoretically analyze its reconstruction property in CNNs. We integrate CRelu
into several state-of-the-art CNN architectures and demonstrate improvement in
their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer
trainable parameters. Our results suggest that better understanding of the
properties of CNNs can lead to significant performance improvement with a
simple modification.

本篇论文提出了一种新的、简单而有效的激活方法，称为 CRelu，并将其与现有的卷积神经网络模型相结合，从而大大提高了它们在 CIFAR-10/100 和 ImageNet 数据集上的性能。