Reasoning is a fundamental component for achieving language understanding.
Among the multiple types of reasoning, conditional reasoning, the ability to
draw different conclusions depending on some condition, has been understudied
in large language models (LLMs). Recent prompting methods, such as chain of
thought, have significantly improved LLMs on reasoning tasks. Nevertheless,
there is still little understanding of what triggers reasoning abilities in
LLMs. We hypothesize that code prompts can trigger conditional reasoning in
LLMs trained on text and code. We propose a chain of prompts that transforms a
natural language problem into code and prompts the LLM with the generated code.
Our experiments find that code prompts exhibit a performance boost between 2.6
and 7.7 points on GPT 3.5 across multiple datasets requiring conditional
reasoning. We then conduct experiments to discover how code prompts elicit
conditional reasoning abilities and through which features. We observe that
prompts need to contain natural language text accompanied by high-quality code
that closely represents the semantics of the instance text. Furthermore, we
show that code prompts are more efficient, requiring fewer demonstrations, and
that they trigger superior state tracking of variables or key entities.

通过将自然语言问题转化为代码并用生成的代码提示进行训练，作者发现代码提示在需要条件推理的多个数据集上提高了 2.6 到 7.7 个 GPT 3.5 的绩效，并且观察到代码提示需要包含自然语言文本和高质量代码，以及其触发更优的变量或关键实例的状态跟踪。

代码提示引发了文本 + 代码模型的条件推理能力

Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs

Recent work has shown that prompting language models with code-like
representations of natural language leads to performance improvements on
structured reasoning tasks. However, such tasks comprise only a small subset of
all natural language tasks. In our work, we seek to answer whether or not
code-prompting is the preferred way of interacting with language models in
general. We compare code and text prompts across three popular GPT models
(davinci, code-davinci-002, and text-davinci-002) on a broader selection of
tasks (e.g., QA, sentiment, summarization) and find that with few exceptions,
code prompts do not consistently outperform text prompts. Furthermore, we show
that the style of code prompt has a large effect on performance for some but
not all tasks and that fine-tuning on text instructions leads to better
relative performance of code prompts.

我们研究了使用编程提示和文本提示对三种流行的 GPT 模型 (davinci、code-davinci-002 和 text-davinci-002) 在更广泛的任务选择 (例如 QA、情感、摘要) 中的表现，结果表明，在很少例外的情况下，编程提示并没有始终优于文本提示。同时，我们还展示了代码提示的样式对某些但不是所有任务的性能有很大的影响，并且微调文本指令可以提高代码提示的相对性能。