While large language models (LLMs) are proficient at question-answering (QA),
the dependencies between their answers and other "beliefs" they may have about
the world are typically unstated, and may even be in conflict. Our goal is to
uncover such dependencies and reduce inconsistencies among them, so that
answers are supported by faithful, system-believed chains of reasoning drawn
from a consistent network of beliefs. Our approach, which we call REFLEX, is to
add a "rational", self-reflecting layer on top of the LLM. First, given a
question, we construct a belief graph using a backward-chaining process to
materialize relevant model "beliefs" (including beliefs about answer
candidates) and the inferential relationships between them. Second, we identify
and minimize contradictions in that graph using a formal constraint reasoner.
We find that REFLEX significantly improves consistency (by 8%-11% absolute)
without harming overall answer accuracy, resulting in answers supported by
faithful chains of reasoning drawn from a more consistent belief system. This
suggests a new style of system architecture, in which an LLM extended with a
rational layer of self-reflection can repair latent inconsistencies within the
LLM alone.

在大型语言模型中添加一个反思层次，通过构建信念图和使用形式化的约束推理器，从而减少其答案与其他 “信念” 的依赖关系和不一致性，REFLEX 方法可以改善一定程度的一致性，同时不降低模型的答案准确性。

具有合理性的语言模型

Language Models with Rationality

Do language models have beliefs about the world? Dennett (1995) famously
argues that even thermostats have beliefs, on the view that a belief is simply
an informational state decoupled from any motivational state. In this paper, we
discuss approaches to detecting when models have beliefs about the world, and
we improve on methods for updating model beliefs to be more truthful, with a
focus on methods based on learned optimizers or hypernetworks. Our main
contributions include: (1) new metrics for evaluating belief-updating methods
that focus on the logical consistency of beliefs, (2) a training objective for
Sequential, Local, and Generalizing model updates (SLAG) that improves the
performance of learned optimizers, and (3) the introduction of the belief
graph, which is a new form of interface with language models that shows the
interdependencies between model beliefs. Our experiments suggest that models
possess belief-like qualities to only a limited extent, but update methods can
both fix incorrect model beliefs and greatly improve their consistency.
Although off-the-shelf optimizers are surprisingly strong belief-updating
baselines, our learned optimizers can outperform them in more difficult
settings than have been considered in past work. Code is available at
this https URL

探讨语言模型是否对世界有信念，介绍检测语言模型是否拥有信念、强化信念更新的方法，其中重点介绍基于学习优化器或超网络的方法，并提出了评估信念更新方法效果的新度量，提出了一种称为 SLAG 的训练目标，并引入了信念图，提供了一种新形式的接口呈现模型信念的内部依赖关系。实验表明，模型具有有限的信念属性，但更新方法可以修正不正确的模型信念，并大大提高其一致性。学习优化器能在比过去工作更困难的环境中胜过现成的优化器。