Efforts to reduce the numerical precision of computations in deep learning
training have yielded systems that aggressively quantize weights and
activations, yet employ wide high-precision accumulators for partial sums in
inner-product operations to preserve the quality of convergence. The absence of
any framework to analyze the precision requirements of partial sum
accumulations results in conservative design choices. This imposes an
upper-bound on the reduction of complexity of multiply-accumulate units. We
present a statistical approach to analyze the impact of reduced accumulation
precision on deep learning training. Observing that a bad choice for
accumulation precision results in loss of information that manifests itself as
a reduction in variance in an ensemble of partial sums, we derive a set of
equations that relate this variance to the length of accumulation and the
minimum number of bits needed for accumulation. We apply our analysis to three
benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet
AlexNet. In each case, with accumulation precision set in accordance with our
proposed equations, the networks successfully converge to the single precision
floating-point baseline. We also show that reducing accumulation precision
further degrades the quality of the trained network, proving that our equations
produce tight bounds. Overall this analysis enables precise tailoring of
computation hardware to the application, yielding area- and power-optimal
systems.

通过统计方法分析深度学习中的累加器准确性，得到了将计算硬件精度进行精确定制的方案，并证明了这种方案可以得到面积和功率最优的系统。