Recent studies suggest that self-reflective prompting can significantly
enhance the reasoning capabilities of Large Language Models (LLMs). However,
the use of external feedback as a stop criterion raises doubts about the true
extent of LLMs' ability to emulate human-like self-reflection. In this paper,
we set out to clarify these capabilities under a more stringent evaluation
setting in which we disallow any kind of external feedback. Our findings under
this setting show a split: while self-reflection enhances performance in
TruthfulQA, it adversely affects results in HotpotQA. We conduct follow-up
analyses to clarify the contributing factors in these patterns, and find that
the influence of self-reflection is impacted both by reliability of accuracy in
models' initial responses, and by overall question difficulty: specifically,
self-reflection shows the most benefit when models are less likely to be
correct initially, and when overall question difficulty is higher. We also find
that self-reflection reduces tendency toward majority voting. Based on our
findings, we propose guidelines for decisions on when to implement
self-reflection. We release the codebase for reproducing our experiments at
this https URL

最近的研究表明，自我反思的提示可以显著提高大型语言模型（LLMs）的推理能力。然而，使用外部反馈作为停止标准对 LLMs 模拟人类自我反思的真正能力提出了疑问。本文旨在在一种更严格的评估环境中澄清这些能力，在该环境中禁止任何形式的外部反馈。我们在这种设定下的研究结果显示出分歧：自我反思在 TruthfulQA 中提高了性能，但对 HotpotQA 的结果产生了不利影响。我们进行了后续分析，以阐明这些模式中的影响因素，并发现自我反思的影响受到模型初始响应的准确性和整体问题难度的影响：具体而言，在模型初始答案不太可能正确和整体问题难度较高时，自我反思显示出最大的益处。我们还发现，自我反思降低了多数投票的趋势。基于我们的发现，我们提出了关于何时实施自我反思的决策指南。我们在此 https URL 发布了复现我们实验的代码库。