Deep neural networks are gaining in popularity as they are used to generate
state-of-the-art results for a variety of computer vision and machine learning
applications. At the same time, these networks have grown in depth and
complexity in order to solve harder problems. Given the limitations in power
budgets dedicated to these networks, the importance of low-power, low-memory
solutions has been stressed in recent years. While a large number of dedicated
hardware using different precisions has recently been proposed, there exists no
comprehensive study of different bit precisions and arithmetic in both inputs
and network parameters. In this work, we address this issue and perform a study
of different bit-precisions in neural networks (from floating-point to
fixed-point, powers of two, and binary). In our evaluation, we consider and
analyze the effect of precision scaling on both network accuracy and hardware
metrics including memory footprint, power and energy consumption, and design
area. We also investigate training-time methodologies to compensate for the
reduction in accuracy due to limited bit precision and demonstrate that in most
cases, precision scaling can deliver significant benefits in design metrics at
the cost of very modest decreases in network accuracy. In addition, we propose
that a small portion of the benefits achieved when using lower precisions can
be forfeited to increase the network size and therefore the accuracy. We
evaluate our experiments, using three well-recognized networks and datasets to
show its generality. We investigate the trade-offs and highlight the benefits
of using lower precisions in terms of energy and memory footprint.

本研究探讨神经网络中不同位精度 (bit-precision) 对网络精度、硬件指标 (存储、功耗、设计面积)、能量消耗等的影响，并提出训练时间方法来弥补位精度降低精度所导致的误差，并表明在降低精度的情况下，可以在设计指标中获得显著的益处，同时提出可以通过增加网络大小来提高精度。

探究精度量化对神经网络准确度和能耗的影响

Understanding the Impact of Precision Quantization on the Accuracy and  Energy of Neural Networks

We explore the redundancy of parameters in deep neural networks by replacing
the conventional linear projection in fully-connected layers with the circulant
projection. The circulant structure substantially reduces memory footprint and
enables the use of the Fast Fourier Transform to speed up the computation.
Considering a fully-connected neural network layer with d input nodes, and d
output nodes, this method improves the time complexity from O(d^2) to O(dlogd)
and space complexity from O(d^2) to O(d). The space savings are particularly
important for modern deep convolutional neural network architectures, where
fully-connected layers typically contain more than 90% of the network
parameters. We further show that the gradient computation and optimization of
the circulant projections can be performed very efficiently. Our experiments on
three standard datasets show that the proposed approach achieves this
significant gain in storage and efficiency with minimal increase in error rate
compared to neural networks with unstructured projections.

本文提出了用循环矩阵代替全连接层中传统的线性投影来探索深度神经网络中参数冗余问题。该循环投影显著降低了存储占用并加速了计算速度，尤其对于现代深度卷积神经网络架构中包含 90% 以上参数的全连接层来说更为重要。作者还展示了该投影的梯度计算和优化可高效完成，并在三个标准数据集上进行了实验证明其存储空间和效率比传统神经网络结构有显著提升，同时误差率仅略微增加。

基于循环投影的深度网络参数冗余探究

An exploration of parameter redundancy in deep networks with circulant  projections

The fully connected layers of a deep convolutional neural network typically
contain over 90% of the network parameters, and consume the majority of the
memory required to store the network parameters. Reducing the number of
parameters while preserving essentially the same predictive performance is
critically important for operating deep neural networks in memory constrained
environments such as GPUs or embedded devices.
In this paper we show how kernel methods, in particular a single Fastfood
layer, can be used to replace all fully connected layers in a deep
convolutional neural network. This novel Fastfood layer is also end-to-end
trainable in conjunction with convolutional layers, allowing us to combine them
into a new architecture, named deep fried convolutional networks, which
substantially reduces the memory footprint of convolutional networks trained on
MNIST and ImageNet with no drop in predictive performance.

本文介绍了使用单个 Fastfood 层代替卷积神经网络中的所有全连接层，并将其与卷积层结合成新的架构，名为深度炸炸卷积网络，从而大幅减少 MNIST 和 ImageNet 训练的卷积网络的内存占用并保持预测性能不变。