Large Language Models (LLMs) have shown superior capability to solve
reasoning problems with programs. While being a promising direction, most of
such frameworks are trained and evaluated in settings with a prior knowledge of
task requirements. However, as LLMs become more capable, it is necessary to
assess their reasoning abilities in more realistic scenarios where many
real-world problems are open-ended with ambiguous scope, and often require
multiple formalisms to solve. To investigate this, we introduce the task of
reasoning in the wild, where an LLM is tasked to solve a reasoning problem of
unknown type by identifying the subproblems and their corresponding formalisms,
and writing a program to solve each subproblem, guided by a tactic. We create a
large tactic-guided trajectory dataset containing detailed solutions to a
diverse set of reasoning problems, ranging from well-defined single-form
reasoning (e.g., math, logic), to ambiguous and hybrid ones (e.g., commonsense,
combined math and logic). This allows us to test various aspects of LLMs
reasoning at the fine-grained level such as the selection and execution of
tactics, and the tendency to take undesired shortcuts. In experiments, we
highlight that existing LLMs fail significantly on problems with ambiguous and
mixed scope, revealing critical limitations and overfitting issues (e.g.
accuracy on GSM8K drops by at least 50\%). We further show the potential of
finetuning a local LLM on the tactic-guided trajectories in achieving better
performance. Project repo is available at
github.com/gblackout/Reason-in-the-Wild

大型语言模型（LLMs）在解决程序推理问题方面显示出了出色的能力。通过引入 “reasoning in the wild” 任务并使用一个包含详细解决方案的大型策略引导轨迹数据集，我们在更真实的场景中评估了 LLMs 的推理能力，揭示了其存在的关键限制。