Multiplications are responsible for most of the computational cost involved in neural network training and inference. Recent research has thus looked for ways to reduce the cost associated with them. Inspired by Mogami (2020), we replace multiplication with a cheap piecewise affine approximation that is achieved by adding the bit representation of the floating point numbers together as integers. We show that transformers can be trained with the resulting modified matrix multiplications on both vision and language tasks with little to no performance impact, and without changes to the training hyperparameters. We further replace all non-linearities in the networks making them fully and jointly piecewise affine in both inputs and weights. Finally, we show that we can eliminate all multiplications in the entire training process, including operations in the forward pass, backward pass and optimizer update, demonstrating the first successful training of modern neural network architectures in a fully multiplication-free fashion.

通过使用廉价的分段仿射逼近方法来代替浮点数乘法并将所有网络非线性替换为分段仿射函数，以实现现代神经网络完全无乘法训练的第一次成功，并展示了对于 transformer 在语言和视觉任务上的性能影响很少的结果修正矩阵乘法训练。

通过分段仿射运算实现硬件高效的Transformer训练