This paper studies the relationship between the surface form of a
mathematical problem and its solvability by large language models. We find that
subtle alterations in the surface form can significantly impact the answer
distribution and the solve rate, exposing the language model's lack of
robustness and sensitivity to the surface form in reasoning through complex
problems. To improve mathematical reasoning performance, we propose
Self-Consistency-over-Paraphrases (SCoP), which diversifies reasoning paths
from specific surface forms of the problem. We evaluate our approach on four
mathematics reasoning benchmarks over three large language models and show that
SCoP improves mathematical reasoning performance over vanilla self-consistency,
particularly for problems initially deemed unsolvable. Finally, we provide
additional experiments and discussion regarding problem difficulty and surface
forms, including cross-model difficulty agreement and paraphrasing
transferability, and Variance of Variations (VOV) for language model
evaluation.

该研究探讨了数学问题的表面形式与其是否能被大型语言模型解决之间的关系，并发现表面形式的微小改变可以显著影响答案分布和解决率，揭示了语言模型在推理复杂问题时缺乏鲁棒性和对表面形式的敏感性。为了提高数学推理性能，研究提出了自洽性超重述（SCoP）方法，该方法通过多样化问题的特定表面形式来多样化推理路径。通过对三个大型语言模型上的四个数学推理基准进行评估，研究表明 SCoP 相对于普通的自洽性方法可以提高数学推理性能，特别是对于最初被认为无法解决的问题。最后，研究提供了关于问题难度和表面形式的额外实验和讨论，包括模型间的难度一致性和重述的可迁移性，以及用于语言模型评估的变异性。