Recent work has demonstrated that the latent spaces of large language models
(LLMs) contain directions predictive of the truth of sentences. Multiple
methods recover such directions and build probes that are described as getting
at a model's "knowledge" or "beliefs". We investigate this phenomenon, looking
closely at the impact of context on the probes. Our experiments establish where
in the LLM the probe's predictions can be described as being conditional on the
preceding (related) sentences. Specifically, we quantify the responsiveness of
the probes to the presence of (negated) supporting and contradicting sentences,
and score the probes on their consistency. We also perform a causal
intervention experiment, investigating whether moving the representation of a
premise along these belief directions influences the position of the hypothesis
along that same direction. We find that the probes we test are generally
context sensitive, but that contexts which should not affect the truth often
still impact the probe outputs. Our experiments show that the type of errors
depend on the layer, the (type of) model, and the kind of data. Finally, our
results suggest that belief directions are (one of the) causal mediators in the
inference process that incorporates in-context information.

我们通过对大型语言模型的隐空间进行研究发现，先前的关于模型的 ' 知识 ' 或' 信念 ' 的研究方法可以基于其隐空间中的方向来构建探测器，而我们的实验结果表明这些探测器的预测可能与前面的（相关）句子有条件关联，并且这种信念方向可以影响推理过程中的真相判断和上下文信息的整合。

语言模型中的真值判断：信念方向是上下文敏感的

Truth-value judgment in language models: belief directions are context  sensitive

As AI technologies are rolled out into healthcare, academia, human resources,
law, and a multitude of other domains, they become de-facto arbiters of truth.
But truth is highly contested, with many different definitions and approaches.
This article discusses the struggle for truth in AI systems and the general
responses to date. It then investigates the production of truth in InstructGPT,
a large language model, highlighting how data harvesting, model architectures,
and social feedback mechanisms weave together disparate understandings of
veracity. It conceptualizes this performance as an operationalization of truth,
where distinct, often conflicting claims are smoothly synthesized and
confidently presented into truth-statements. We argue that these same logics
and inconsistencies play out in Instruct's successor, ChatGPT, reiterating
truth as a non-trivial problem. We suggest that enriching sociality and
thickening "reality" are two promising vectors for enhancing the
truth-evaluating capacities of future language models. We conclude, however, by
stepping back to consider AI truth-telling as a social practice: what kind of
"truth" do we as listeners desire?

本文讨论了 AI 系统中真理的斗争及其对 InstructGPT 的性能进行了调查，强调了数据收集、模型架构和社会反馈机制如何将各种看法巧妙地综合并自信地呈现为真实陈述，提出了增强未来语言模型真理评估能力的两个可行方向，即丰富社交性和增强 “现实”。