Mathematical capabilities were previously believed to emerge in common
language models only at a very large scale or require extensive math-related
pre-training. This paper shows that the LLaMA-2 7B model with common
pre-training already exhibits strong mathematical abilities, as evidenced by
its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks,
respectively, when selecting the best response from 256 random generations. The
primary issue with the current base model is the difficulty in consistently
eliciting its inherent mathematical capabilities. Notably, the accuracy for the
first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks,
respectively. We find that simply scaling up the SFT data can significantly
enhance the reliability of generating correct answers. However, the potential
for extensive scaling is constrained by the scarcity of publicly available math
questions. To overcome this limitation, we employ synthetic data, which proves
to be nearly as effective as real data and shows no clear saturation when
scaled up to approximately one million samples. This straightforward approach
achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B
models, surpassing previous models by 14.2% and 20.8%, respectively. We also
provide insights into scaling behaviors across different reasoning complexities
and error types.

LLaMA-2 7B 模型通过简单的方法扩展数据样本，证明了其出色的数学能力及可靠性，适用于 GSM8K 和 MATH 基准测试，并提供了关于不同推理复杂性和错误类型的扩展行为的见解。