This paper presents the first comprehensive empirical study demonstrating the
efficacy of the Brain Floating Point (BFLOAT16) half-precision format for Deep
Learning training across image classification, speech recognition, language
modeling, generative networks and industrial recommendation systems. BFLOAT16
is attractive for Deep Learning training for two reasons: the range of values
it can represent is the same as that of IEEE 754 floating-point format (FP32)
and conversion to/from FP32 is simple. Maintaining the same range as FP32 is
important to ensure that no hyper-parameter tuning is required for convergence;
e.g., IEEE 754 compliant half-precision floating point (FP16) requires
hyper-parameter tuning. In this paper, we discuss the flow of tensors and
various key operations in mixed precision training, and delve into details of
operations, such as the rounding modes for converting FP32 tensors to BFLOAT16.
We have implemented a method to emulate BFLOAT16 operations in Tensorflow,
Caffe2, IntelCaffe, and Neon for our experiments. Our results show that deep
learning training using BFLOAT16 tensors achieves the same state-of-the-art
(SOTA) results across domains as FP32 tensors in the same number of iterations
and with no changes to hyper-parameters.

本文研究了使用 Brain Floating Point（BFLOAT16）半精度格式进行深度学习训练的有效性，并实现了在 Tensorflow、Caffe2、IntelCaffe 和 Neon 中进行 BFLOAT16 操作的方法。结果表明，使用 BFLOAT16 张量进行深度学习培训，在达到同样的状态 - 最先进的结果并不需要调整超参数。

BFLOAT16 在深度学习训练中的研究

A Study of BFLOAT16 for Deep Learning Training

Efforts to reduce the numerical precision of computations in deep learning
training have yielded systems that aggressively quantize weights and
activations, yet employ wide high-precision accumulators for partial sums in
inner-product operations to preserve the quality of convergence. The absence of
any framework to analyze the precision requirements of partial sum
accumulations results in conservative design choices. This imposes an
upper-bound on the reduction of complexity of multiply-accumulate units. We
present a statistical approach to analyze the impact of reduced accumulation
precision on deep learning training. Observing that a bad choice for
accumulation precision results in loss of information that manifests itself as
a reduction in variance in an ensemble of partial sums, we derive a set of
equations that relate this variance to the length of accumulation and the
minimum number of bits needed for accumulation. We apply our analysis to three
benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet
AlexNet. In each case, with accumulation precision set in accordance with our
proposed equations, the networks successfully converge to the single precision
floating-point baseline. We also show that reducing accumulation precision
further degrades the quality of the trained network, proving that our equations
produce tight bounds. Overall this analysis enables precise tailoring of
computation hardware to the application, yielding area- and power-optimal
systems.

通过统计方法分析深度学习中的累加器准确性，得到了将计算硬件精度进行精确定制的方案，并证明了这种方案可以得到面积和功率最优的系统。