Human cognition exhibits systematic compositionality, the algebraic ability
to generate infinite novel combinations from finite learned components, which
is the key to understanding and reasoning about complex logic. In this work, we
investigate the compositionality of large language models (LLMs) in
mathematical reasoning. Specifically, we construct a new dataset
\textsc{MathTrap}\footnotemark[3] by introducing carefully designed logical
traps into the problem descriptions of MATH and GSM8k. Since problems with
logical flaws are quite rare in the real world, these represent ``unseen''
cases to LLMs. Solving these requires the models to systematically compose (1)
the mathematical knowledge involved in the original problems with (2) knowledge
related to the introduced traps. Our experiments show that while LLMs possess
both components of requisite knowledge, they do not \textbf{spontaneously}
combine them to handle these novel cases. We explore several methods to
mitigate this deficiency, such as natural language prompts, few-shot
demonstrations, and fine-tuning. We find that LLMs' performance can be
\textbf{passively} improved through the above external intervention. Overall,
systematic compositionality remains an open challenge for large language
models.

大型语言模型在数学推理方面研究了系统组合性，通过引入精心设计的逻辑漏洞陷阱构建了一个新的数据集 MathTrap，发现虽然大型语言模型具备所需知识的两个组成部分，但它们不会自发地组合起来处理这些新问题，通过自然语言提示、少示范演示和微调等多种方法可以减轻这种缺陷，但系统的组合性仍然是大型语言模型面临的一个未解决挑战。