In utilizing large language models (LLMs) for mathematical reasoning,
addressing the errors in the reasoning and calculation present in the generated
text by LLMs is a crucial challenge. In this paper, we propose a novel
framework that integrates the Chain-of-Thought (CoT) method with an external
tool (Python REPL). We discovered that by prompting LLMs to generate structured
text in XML-like markup language, we could seamlessly integrate CoT and the
external tool and control the undesired behaviors of LLMs. With our approach,
LLMs can utilize Python computation to rectify errors within CoT. We applied
our method to ChatGPT (GPT-3.5) to solve challenging mathematical problems and
demonstrated that combining CoT and Python REPL through the markup language
enhances the reasoning capability of LLMs. Our approach enables LLMs to write
the markup language and perform advanced mathematical reasoning using only
zero-shot prompting.

利用大型语言模型（LLMs）进行数学推理时，整合 Chain-of-Thought 方法与外部工具（Python REPL）通过标记语言无缝结合，使 LLMs 可以纠正推理中的错误，提高推理能力。

LPML: 数学推理的 LLM-Prompting 标记语言

LPML: LLM-Prompting Markup Language for Mathematical Reasoning

Analogical reasoning is essential for human cognition, allowing us to
comprehend new concepts by relating them to familiar ones based on common
relational structures. Previous work mainly focuses on word analogies, which do
not fully represent the analogical reasoning ability of language models (LMs)
aligning with humans. This paper first examines analogy prompting for large
language models (LLMs) in scientific question-answering tasks. Then we discover
that LLMs tend to ignore relational structures when performing word analogies,
casting doubt on their utility for evaluating analogical reasoning. For better
evaluation aligning with humans, we propose an analogical structure abduction
task based on cognitive psychology, which aims to abduct structures between two
systems to establish an analogy. Then we create a benchmark of scientific
analogical reasoning with structure abduction, SCAR, consisting of 400
scientific analogies across 13 domains for this task. Empirical results reveal
that LLMs struggle with this task, but the Chain-of-Thought (CoT) method with
background knowledge and explanations can improve their capability.

本文介绍了关于大型语言模型与人类类比推理的一些发现和研究，并提出了一种基于认知心理学的类比结构推断任务以及用于评估类比推理能力的 SCAR 基准测试集，实验结果表明，结合背景知识和解释的 CoT 方法可以显著提高 LLMs 在该任务上的表现。