A deep neural network (DNN) with piecewise linear activations can partition
the input space into numerous small linear regions, where different linear
functions are fitted. It is believed that the number of these regions
represents the expressivity of the DNN. This paper provides a novel and
meticulous perspective to look into DNNs: Instead of just counting the number
of the linear regions, we study their local properties, such as the inspheres,
the directions of the corresponding hyperplanes, the decision boundaries, and
the relevance of the surrounding regions. We empirically observed that
different optimization techniques lead to completely different linear regions,
even though they result in similar classification accuracies. We hope our study
can inspire the design of novel optimization techniques, and help discover and
analyze the behaviors of DNNs.

通过局部属性的研究，探究了 DNN 的表达能力和优化技术对其决策边界的影响，并希望这一研究能够启示新型优化技术的设计和发现 DNN 的行为模式。

深度神经网络中线性区域特性的实证研究

Empirical Studies on the Properties of Linear Regions in Deep Neural  Networks

We present a new, unifying approach following some recent developments on the
complexity of neural networks with piecewise linear activations. We treat
neural network layers with piecewise linear activations as tropical
polynomials, which generalize polynomials in the so-called $(\max, +)$ or
tropical algebra, with possibly real-valued exponents. Motivated by the
discussion in (arXiv:1402.1869), this approach enables us to refine their upper
bounds on linear regions of layers with ReLU or leaky ReLU activations to
$\min\left\{ 2^m, \sum_{j=0}^n \binom{m}{j} \right\}$, where $n, m$ are the
number of inputs and outputs, respectively. Additionally, we recover their
upper bounds on maxout layers. Our work follows a novel path, exclusively under
the lens of tropical geometry, which is independent of the improvements
reported in (arXiv:1611.01491, arXiv:1711.02114). Finally, we present a
geometric approach for effective counting of linear regions using random
sampling in order to avoid the computational overhead of exact counting
approaches

本研究使用热带代数中 generalization 的概念对神经网络中的分段线性激活函数进行统一且独特的处理，将神经网络层看做是一个 tropical polynomial，进一步优化计算线性区域的上界，并提出了一种使用随机抽样有效计算线性区域数量的几何方法。

分段线性激活函数的神经网络热带方法

A Tropical Approach to Neural Networks with Piecewise Linear Activations

We study the complexity of functions computable by deep feedforward neural
networks with piecewise linear activations in terms of the symmetries and the
number of linear regions that they have. Deep networks are able to sequentially
map portions of each layer's input-space to the same output. In this way, deep
models compute functions that react equally to complicated patterns of
different inputs. The compositional structure of these functions enables them
to re-use pieces of computation exponentially often in terms of the network's
depth. This paper investigates the complexity of such compositional maps and
contributes new theoretical results regarding the advantage of depth for neural
networks with piecewise linear activation functions. In particular, our
analysis is not specific to a single family of models, and as an example, we
employ it for rectifier and maxout networks. We improve complexity bounds from
pre-existing work and investigate the behavior of units in higher layers.

研究神经网络的深度和节点的线性区分性对于复杂计算的影响，并在对多个模型类型的分析中提出新的关于深度优势的理论结果，同时研究了高层次单元的行为。