The ability (and inability) of large language models (LLMs) to perform
arithmetic tasks has been the subject of much theoretical and practical debate.
We show that LLMs are frequently able to correctly and confidently predict the
first digit of n-digit by m-digit multiplication tasks without using chain of
thought reasoning, despite these tasks require compounding operations to solve.
Simultaneously, LLMs in practice often fail to correctly or confidently predict
the last digit of an n-digit by m-digit multiplication, a task equivalent to
1-digit by 1-digit multiplication which can be easily learned or memorized. We
show that the latter task can be solved more robustly when the LLM is
conditioned on all of the correct higher-order digits, which on average
increases the confidence of the correct last digit on 5-digit by 5-digit
multiplication tasks using Llama 2-13B by over 230% (0.13 to 0.43) and
Mistral-7B by 150% (0.22 to 0.55).

大型语言模型在进行算术任务中对多位数乘法的可信准确预测能力较强，而对单位数乘法可信准确预测的能力较差，但在给出正确的高位数字作为条件下，后者任务的可信准确预测能力可以大幅提升。

语言模型轻松完成难算术任务，但几乎不能完成简单算术任务

Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy  Arithmetic Tasks

We introduce Goat, a fine-tuned LLaMA model that significantly outperforms
GPT-4 on a range of arithmetic tasks. Fine-tuned on a synthetically generated
dataset, Goat achieves state-of-the-art performance on BIG-bench arithmetic
sub-task. In particular, the zero-shot Goat-7B matches or even surpasses the
accuracy achieved by the few-shot PaLM-540B. Surprisingly, Goat can achieve
near-perfect accuracy on large-number addition and subtraction through
supervised fine-tuning only, which is almost impossible with previous
pretrained language models, such as Bloom, OPT, GPT-NeoX, etc. We attribute
Goat's exceptional performance to LLaMA's consistent tokenization of numbers.
To tackle more challenging tasks like large-number multiplication and division,
we propose an approach that classifies tasks based on their learnability, and
subsequently decomposes unlearnable tasks, such as multi-digit multiplication
and division, into a series of learnable tasks by leveraging basic arithmetic
principles. We thoroughly examine the performance of our model, offering a
comprehensive evaluation of the effectiveness of our proposed decomposition
steps. Additionally, Goat-7B can be easily trained using LoRA on a 24GB VRAM
GPU, facilitating reproducibility for other researchers. We release our model,
dataset, and the Python script for dataset generation.

Goat，用 LLaMA 模型细调后，在一系列算术任务中显著优于 GPT-4，特别是在大数字加法和减法方面表现出了非常高的准确性。 对于更具挑战性的任务，如大数字乘法和除法，我们提出了一种根据其可学习性分类任务的方法，并通过利用基本算术原则将不可学习的任务（如多位数乘法和除法）分解为一系列可学习的任务。