CoT (Chain-of-Thought) is a way to solve reasoning problems for LLMs .
Recently, many researches appear for improving the CoT capability of LLMs. In
this work, we also proposed Olapa-MCoT, which is a LLMs based on llama2-13B PLM
for finetuning and alignment learning. During the alignment training, we
proposed the SimRRHF algorithm and Incorrect Data Relearning and mainly focused
on optimizing the Chinese mathematical reasoning ability of Olapa-MCoT. The
experiment achieved significant results, with the accuracy of Chinese
mathematical reasoning up to 50%, 36% rise compared to llama2-13B. In addition,
the accuracy of English reasoning ability also increased by nearly 4%.

这篇论文提出了基于 llama2-13B PLM 的深度语言模型 Olapa-MCoT，通过使用 SimRRHF 算法和错误数据再学习，主要优化了 Olapa-MCoT 的中文数学推理能力，实验结果表明中文数学推理准确率高达 50%，相比于 llama2-13B 提升了 36%；此外，英语推理能力的准确率也提高了近 4%。

Olapa-MCoT：提升 LLMs 的中文数学推理能力

Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of  LLMs

Reasoning is a cognitive process of using evidence to reach a sound
conclusion. The reasoning capability is essential for large language models
(LLMs) to serve as the brain of the artificial general intelligence agent.
Recent studies reveal that fine-tuning LLMs on data with the chain of thought
(COT) reasoning process can significantly enhance their reasoning capabilities.
However, we find that the fine-tuned LLMs suffer from an \textit{Assessment
Misalignment} problem, i.e., they frequently assign higher scores to subpar
COTs, leading to potential limitations in their reasoning abilities. To address
this problem, we introduce an \textit{Alignment Fine-Tuning (AFT)} paradigm,
which involves three steps: 1) fine-tuning LLMs with COT training data; 2)
generating multiple COT responses for each question, and categorizing them into
positive and negative ones based on whether they achieve the correct answer; 3)
calibrating the scores of positive and negative responses given by LLMs with a
novel constraint alignment loss. Specifically, the constraint alignment loss
has two objectives: a) Alignment, which guarantees that positive scores surpass
negative scores to encourage answers with high-quality COTs; b) Constraint,
which keeps the negative scores confined to a reasonable range to prevent the
model degradation. Beyond just the binary positive and negative feedback, the
constraint alignment loss can be seamlessly adapted to the ranking situations
when ranking feedback is accessible. Furthermore, we also delve deeply into
recent ranking-based alignment methods, such as DPO, RRHF, and PRO, and
discover that the constraint, which has been overlooked by these approaches, is
also crucial for their performance. Extensive experiments on four reasoning
benchmarks with both binary and ranking feedback demonstrate the effectiveness
of AFT.

Reasoning is essential for large language models, and the Alignment Fine-Tuning (AFT) paradigm addresses the Assessment Misalignment problem in fine-tuned models, improving their reasoning abilities by calibrating responses and introducing constraint alignment loss.

通过对齐使大型语言模型成为更好的推理者

Making Large Language Models Better Reasoners with Alignment

Chain-of-Thought Prompting (CoT) reinforces the reasoning capabilities of
Large Language Models (LLMs) through the generation of intermediate rationales.
However, these enhancements predominantly benefit large-scale models, leaving
small LMs without significant performance improvements when directly applying
CoT. Despite the advanced reasoning capabilities of LLMs, CoT relies primarily
on their pre-trained internal knowledge. The external knowledge that is
previously unknown to the model remains unexploited. This omission becomes
pronounced in tasks such as stance detection, where the external background
knowledge plays a pivotal role. Additionally, the large-scale architecture of
LLMs inevitably present efficiency challenges during deployment. To address
these challenges, we introduce the Ladder-of-Thought (LoT) for stance
detection. Grounded in a dual-phase Cascaded Optimization framework, LoT
directs the model to incorporate high-quality external knowledge, enhancing the
intermediate rationales it generates. These bolstered rationales subsequently
serve as the foundation for more precise predictions - akin to how a ladder
facilitates reaching elevated goals. LoT achieves a balance between efficiency
and accuracy, making it an adaptable and efficient framework for stance
detection. Our empirical evaluations underscore LoT's effectiveness, marking a
16% improvement over ChatGPT and a 10% enhancement compared to ChatGPT with
CoT.

通过引入 Ladder-of-Thought (LoT) 来改进 ChatGPT 和 CoT 模型，在提高中间推理过程和使用外部知识的基础上，在立场检测任务中实现了 16% 的改进。

思维之阶梯：利用知识提升立场检测

Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection

Chain-of-thought prompting (CoT) advances the reasoning abilities of large
language models (LLMs) and achieves superior performance in arithmetic,
commonsense, and symbolic reasoning tasks. However, most CoT studies rely on
carefully designed human-annotated rational chains to prompt the language
model, which poses challenges for real-world applications where labeled
training data is available without human-annotated rational chains. This
creates barriers to applications of CoT prompting to these general tasks. This
paper proposes a new strategy, Automate-CoT (Automatic Prompt Augmentation and
Selection with Chain-of-Thought), that can bypass human engineering of CoTs by
automatically augmenting rational chains from a small labeled dataset, and then
pruning low-quality chains to construct a candidate pool of machine-generated
rationale chains based on the labels. Finally, it selects the optimal
combination of several rationale chains from the pool for CoT prompting by
employing a variance-reduced policy gradient strategy to estimate the
significance of each example in a black-box language model. Automate-CoT
enables a quick adaptation of the CoT technique to different tasks.
Experimental results demonstrate the effectiveness of our method, where
state-of-the-art results are achieved on arithmetic reasoning (+2.7\%),
commonsense reasoning (+3.4\%), symbolic reasoning (+3.2\%), and non-reasoning
tasks (+2.5\%). Our code will be available at
this https URL.

本文提出了一种使用 Automate-CoT 自动扩展和选择合理链来推动 CoT 的方法，该方法通过构建候选池从标签生成的机器理性链中选择多个理性链的最佳组合，以推进大型语言模型的推理能力，并在算术推理、常识推理、符号推理和非推理任务中取得了最先进的结果。