While the emergence of powerful language models along with Chain-of-thought prompting has made automation more and more omnipresent, it sometimes demonstrates its weakness in long-term or multi-step logical reasoning. For example, users don't always get desirable answers for complex mathematical problems without human involvement. Against this background, we present the Manual Correction System (MCS) -- a human-in-the-loop system enhanced by Chain-of-Thought prompting, which explores how manual correction of sub-logics in rationales can improve LLM's reasoning performance. Moving one step forward, considering a system with human-in-the-loop involves more than having humans improve performance but also controlling the cost. Therefore, we post a Cost-utility Analysis Model for Human-in-the-Loop systems (CAMLOP) based on classical economics theory to analyze, quantify and balance the utility and the corresponding cost. We conduct experiments of MCS and CAMLOP with twelve datasets. A significant advantage w.r.t cost and utility proves its superiority over strong baselines.

本文介绍了一种人在循环中的系统，Manual Correction System (MCS)，使用Chain-of-Thought提示增强LLM的推理表现，并且提出了基于古典经济理论的Cost-Utility分析模型(CAMLOP)来分析、量化和平衡效用和相应成本。作者进行了MCS和CAMLOP实验，并与12个数据集的强基线进行比较，结果证明其超越了强基线，并具有成本和效用的显着优势。

链式思考中的人类参与