Large Language Models (LLMs) have shown superior capability to solve
reasoning problems with programs. While being a promising direction, most of
such frameworks are trained and evaluated in settings with a prior knowledge of
task requirements. However, as LLMs become more capable, it is necessary to
assess their reasoning abilities in more realistic scenarios where many
real-world problems are open-ended with ambiguous scope, and often require
multiple formalisms to solve. To investigate this, we introduce the task of
reasoning in the wild, where an LLM is tasked to solve a reasoning problem of
unknown type by identifying the subproblems and their corresponding formalisms,
and writing a program to solve each subproblem, guided by a tactic. We create a
large tactic-guided trajectory dataset containing detailed solutions to a
diverse set of reasoning problems, ranging from well-defined single-form
reasoning (e.g., math, logic), to ambiguous and hybrid ones (e.g., commonsense,
combined math and logic). This allows us to test various aspects of LLMs
reasoning at the fine-grained level such as the selection and execution of
tactics, and the tendency to take undesired shortcuts. In experiments, we
highlight that existing LLMs fail significantly on problems with ambiguous and
mixed scope, revealing critical limitations and overfitting issues (e.g.
accuracy on GSM8K drops by at least 50\%). We further show the potential of
finetuning a local LLM on the tactic-guided trajectories in achieving better
performance. Project repo is available at
github.com/gblackout/Reason-in-the-Wild

大型语言模型（LLMs）在解决程序推理问题方面显示出了出色的能力。通过引入 “reasoning in the wild” 任务并使用一个包含详细解决方案的大型策略引导轨迹数据集，我们在更真实的场景中评估了 LLMs 的推理能力，揭示了其存在的关键限制。

LLMs 能否通过程序在复杂环境进行推理？

Can LLMs Reason in the Wild with Programs?

Scenario-based testing of automated driving functions has become a promising
method to reduce time and cost compared to real-world testing. In
scenario-based testing automated functions are evaluated in a set of
pre-defined scenarios. These scenarios provide information about vehicle
behaviors, environmental conditions, or road characteristics using parameters.
To create realistic scenarios, parameters and parameter dependencies have to be
fitted utilizing real-world data. However, due to the large variety of
intersections and movement constellations found in reality, data may not be
available for certain scenarios. This paper proposes a methodology to
systematically analyze relations between parameters of scenarios. Bayesian
networks are utilized to analyze causal dependencies in order to decrease the
amount of required data and to transfer causal patterns creating unseen
scenarios. Thereby, infrastructural influences on movement patterns are
investigated to generate realistic scenarios on unobserved intersections. For
evaluation, scenarios and underlying parameters are extracted from the inD
dataset. Movement patterns are estimated, transferred and checked against
recorded data from those initially unseen intersections.

通过利用贝叶斯网络分析各个参数的关系，本文提出了一种系统的方法来降低对数据的需求量，并在未观测到的交叉口生成真实的场景，以此评估自动驾驶功能的场景测试。