Existing research predominantly focuses on developing powerful language
learning models (LLMs) for mathematical reasoning within monolingual languages,
with few explorations in preserving efficacy in a multilingual context. To
bridge this gap, this paper pioneers exploring and training powerful
Multilingual Math Reasoning (xMR) LLMs. Firstly, by utilizing translation, we
construct the first multilingual math reasoning instruction dataset,
MGSM8KInstruct, encompassing ten distinct languages, thus addressing the issue
of training data scarcity in xMR tasks. Based on the collected dataset, we
propose different training strategies to build powerful xMR LLMs, named
MathOctopus, notably outperform conventional open-source LLMs and exhibit
superiority over ChatGPT in few-shot scenarios. Notably, MathOctopus-13B
reaches 47.6% accuracy which exceeds ChatGPT 46.3% on MGSM testset. Beyond
remarkable results, we unearth several pivotal observations and insights from
extensive experiments: (1) When extending the rejection sampling strategy to
the multilingual context, it proves effective for model performances, albeit
limited. (2) Employing parallel corpora for math Supervised Fine-Tuning (SFT)
across multiple languages not only significantly enhances model performance
multilingually but also elevates their monolingual performance. This indicates
that crafting multilingual corpora can be regarded as a vital strategy for
enhancing model performance in a specific language, especially in mathematical
reasoning tasks. For instance, MathOctopus-7B improves its counterparts that
trained on English from 42.2% to 50.8% on GSM8K testset.

本文首次探索和训练强大的多语种数学推理语言学习模型，通过利用翻译构建了首个多语种数学推理指令数据集 MGSM8KInstruct，提出不同的训练策略，构建了名为 MathOctopus 的强大的多语种数学推理模型，其在少样本情况下表现优于传统开源模型和 ChatGPT。