Nowadays, the number of layers and of neurons in each layer of a deep network
are typically set manually. While very deep and wide networks have proven
effective in general, they come at a high memory and computation cost, thus
making them impractical for constrained platforms. These networks, however, are
known to have many redundant parameters, and could thus, in principle, be
replaced by more compact architectures. In this paper, we introduce an approach
to automatically determining the number of neurons in each layer of a deep
network during learning. To this end, we propose to make use of structured
sparsity during learning. More precisely, we use a group sparsity regularizer
on the parameters of the network, where each group is defined to act on a
single neuron. Starting from an overcomplete network, we show that our approach
can reduce the number of parameters by up to 80\% while retaining or even
improving the network accuracy.

使用结构稀疏性和组稀疏正则化对深度网络进行参数优化和削减，达到最高 80％的参数削减率并保持或提高网络精度。

深度网络的神经元数量学习

Learning the Number of Neurons in Deep Networks

Large-scale artificial neural networks have shown significant promise in
addressing a wide range of classification and recognition applications.
However, their large computational requirements stretch the capabilities of
computing platforms. The fundamental components of these neural networks are
the neurons and its synapses. The core of a digital hardware neuron consists of
multiplier, accumulator and activation function. Multipliers consume most of
the processing energy in the digital neurons, and thereby in the hardware
implementations of artificial neural networks. We propose an approximate
multiplier that utilizes the notion of computation sharing and exploits error
resilience of neural network applications to achieve improved energy
consumption. We also propose Multiplier-less Artificial Neuron (MAN) for even
larger improvement in energy consumption and adapt the training process to
ensure minimal degradation in accuracy. We evaluated the proposed design on 5
recognition applications. The results show, 35% and 60% reduction in energy
consumption, for neuron sizes of 8 bits and 12 bits, respectively, with a
maximum of ~2.83% loss in network accuracy, compared to a conventional neuron
implementation. We also achieve 37% and 62% reduction in area for a neuron size
of 8 bits and 12 bits, respectively, under iso-speed conditions.

提出了一种利用计算共享和神经网络应用程序的误差容忍性来实现能源消耗改进的近似乘法器和无乘法器人工神经元，并采用最小精度损失的训练方法。实验结果表明，在与传统神经元实现比较下，8 位和 12 位神经元大小的能源消耗分别降低了 35% 和 60%，最大网络精度损失约为 2.83%。此外，在 iso-speed 条件下，8 位和 12 位的神经元大小分别减少了 37% 和 62% 的面积。