Transformers, central to the successes in modern Natural Language Processing,
often falter on arithmetic tasks despite their vast capabilities --which
paradoxically include remarkable coding abilities. We observe that a crucial
challenge is their naive reliance on positional information to solve arithmetic
problems with a small number of digits, leading to poor performance on larger
numbers. Herein, we delve deeper into the role of positional encoding, and
propose several ways to fix the issue, either by modifying the positional
encoding directly, or by modifying the representation of the arithmetic task to
leverage standard positional encoding differently. We investigate the value of
these modifications for three tasks: (i) classical multiplication, (ii) length
extrapolation in addition, and (iii) addition in natural language context. For
(i) we train a small model on a small dataset (100M parameters and 300k
samples) with remarkable aptitude in (direct, no scratchpad) 15 digits
multiplication and essentially perfect up to 12 digits, while usual training in
this context would give a model failing at 4 digits multiplication. In the
experiments on addition, we use a mere 120k samples to demonstrate: for (ii)
extrapolation from 10 digits to testing on 12 digits numbers while usual
training would have no extrapolation, and for (iii) almost perfect accuracy up
to 5 digits while usual training would be correct only up to 3 digits (which is
essentially memorization with a training set of 120k samples).

通过修改位置编码或者调整算术任务的表示方式来解决使用位置信息解决小数字算术问题时性能较差的问题，并在乘法和加法任务中展示了改进的效果。

位置描述对于 Transformer 算术很重要

Positional Description Matters for Transformers Arithmetic

Previous studies have typically assumed that large language models are unable
to accurately perform arithmetic operations, particularly multiplication of >8
digits, and operations involving decimals and fractions, without the use of
calculator tools. This paper aims to challenge this misconception. With
sufficient training data, a 2 billion-parameter language model can accurately
perform multi-digit arithmetic operations with almost 100% accuracy without
data leakage, significantly surpassing GPT-4 (whose multi-digit multiplication
accuracy is only 4.3%). We also demonstrate that our MathGLM, fine-tuned from
GLM-10B on a dataset with additional multi-step arithmetic operations and math
problems described in text, achieves similar performance to GPT-4 on a
5,000-samples Chinese math problem test set.

大型语言模型可以准确执行包括多位数字乘法、小数和分数在内的算术运算，并且能够在无数据泄露情况下实现几乎 100% 的准确度，超越了 GPT-4。此外，我们通过在包含多步骤算术运算和文本描述的数据集上对 GLM-10B 进行微调，开发出的 MathGLM 在 5000 个样本的中文数学问题测试集上展现了与 GPT-4 相似的性能。

GPT 能够在没有计算器的情况下解决数学问题

GPT Can Solve Mathematical Problems Without a Calculator

Multiplication is indispensable and is one of the core operations in many
modern applications including signal processing and neural networks.
Conventional right-to-left (RL) multiplier extensively contributes to the power
consumption, area utilization and critical path delay in such applications.
This paper proposes a low latency multiplier based on online or left-to-right
(LR) arithmetic which can increase throughput and reduce latency by digit-level
pipelining. Online arithmetic enables overlapping successive operations
regardless of data dependency because of the most significant digit first mode
of operation. To produce most significant digit first, it uses redundant number
system and we can have a carry-free addition, therefore, the delay of the
arithmetic operation is independent of operand bit width. The operations are
performed digit by digit serially from left to right which allows gradual
increase in the slice activities making it suitable for implementation on
reconfigurable devices. Serial nature of the online algorithm and gradual
increment/decrement of active slices minimize the interconnects and signal
activities resulting in overall reduction of area and power consumption. We
present online multipliers with; both inputs in serial, and one in serial and
one in parallel. Pipelined and non-pipelined designs of the proposed
multipliers have been synthesized with GSCL 45nm technology on Synopsys Design
Compiler. Thorough comparative analysis has been performed using widely used
performance metrics. The results show that the proposed online multipliers
outperform the RL multipliers.

本研究提出基于在线算法或从左到右算法的低延迟乘法器，利用冗余数字系统产生最高位优先，并通过数字级管道衔接进行连续操作，从而优化性能并减少功耗和面积。

内积数组低活动在线低延迟乘法器

Low-Latency Online Multiplier with Reduced Activities and Minimized  Interconnect for Inner Product Arrays

Multiplication (e.g., convolution) is arguably a cornerstone of modern deep
neural networks (DNNs). However, intensive multiplications cause expensive
resource costs that challenge DNNs' deployment on resource-constrained edge
devices, driving several attempts for multiplication-less deep networks. This
paper presented ShiftAddNet, whose main inspiration is drawn from a common
practice in energy-efficient hardware implementation, that is, multiplication
can be instead performed with additions and logical bit-shifts. We leverage
this idea to explicitly parameterize deep networks in this way, yielding a new
type of deep network that involves only bit-shift and additive weight layers.
This hardware-inspired ShiftAddNet immediately leads to both energy-efficient
inference and training, without compromising the expressive capacity compared
to standard DNNs. The two complementary operation types (bit-shift and add)
additionally enable finer-grained control of the model's learning capacity,
leading to more flexible trade-off between accuracy and (training) efficiency,
as well as improved robustness to quantization and pruning. We conduct
extensive experiments and ablation studies, all backed up by our FPGA-based
ShiftAddNet implementation and energy measurements. Compared to existing DNNs
or other multiplication-less models, ShiftAddNet aggressively reduces over 80%
hardware-quantified energy cost of DNNs training and inference, while offering
comparable or better accuracies. Codes and pre-trained models are available at
this https URL

ShiftAddNet 是一种硬件启发式的深度神经网络，通过只使用位移和加权层替代传统的乘法运算来实现深度网络的显式参数化，从而获得与标准深度神经网络相当的表达能力和更加灵活的精度和效率之间的权衡，并且可以与量化和修剪等技术结合，实现在训练和推理中更加节能高效的性能。