Large language models (LLMs) have showcased impressive multilingual machine
translation ability. However, unlike encoder-decoder style models, decoder-only
LLMs lack an explicit alignment between source and target contexts. Analyzing
contribution scores during generation processes revealed that LLMs can be
biased towards previously generated tokens over corresponding source tokens,
leading to unfaithful translations. To address this issue, we propose to
encourage LLMs to pay more attention to the source context from both source and
target perspectives in zeroshot prompting: 1) adjust source context attention
weights; 2) suppress irrelevant target prefix influence; Additionally, we
propose 3) avoiding over-reliance on the target prefix in instruction tuning.
Experimental results from both human-collected unfaithfulness test sets
focusing on LLM-generated unfaithful translations and general test sets, verify
our methods' effectiveness across multiple language pairs. Further human
evaluation shows our method's efficacy in reducing hallucinatory translations
and facilitating faithful translation generation.

在这篇研究中，我们提出了一种方法来解决大型语言模型在翻译过程中因未对齐问题而导致的不准确翻译的困扰，该方法通过调整源语言和目标语言的注意力权重、压制无关目标前缀的影响以及避免过度依赖目标前缀来改善翻译质量，实验证明该方法在多种语言对上的效果显著，有效地减少幻觉式翻译，并促进准确翻译的生成。

更加注意源语言上下文：减轻大型语言模型带来的不忠译

Paying More Attention to Source Context: Mitigating Unfaithful  Translations from Large Language Model

Large language models (LLMs) improve their performance in downstream tasks
when they generate Chain of Thought reasoning text before producing an answer.
Our research investigates how LLMs recover from errors in Chain of Thought,
reaching the correct final answer despite mistakes in the reasoning text.
Through analysis of these error recovery behaviors, we find evidence for
unfaithfulness in Chain of Thought, but we also identify many clear examples of
faithful error recovery behaviors. We identify factors that shift LLM recovery
behavior: LLMs recover more frequently from obvious errors and in contexts that
provide more evidence for the correct answer. However, unfaithful recoveries
show the opposite behavior, occurring more frequently for more difficult error
positions. Our results indicate that there are distinct mechanisms driving
faithful and unfaithful error recoveries. Our results challenge the view that
LLM reasoning is a uniform, coherent process.

大型语言模型（LLMs）在输出答案之前生成思维链推理文本可以提高其在下游任务中的性能。我们的研究调查了 LLMs 在思维链中出现错误时如何还原并最终得出正确答案，我们发现存在思维链不忠实的证据，但我们也找到了许多明确的忠实错误还原行为。我们确定了影响 LLM 还原行为的因素：LLMs 在明显的错误和能提供更多正确答案证据的情境中更频繁地还原。然而，不忠实的还原行为则表现相反，在更困难的错误位置更频繁地发生。我们的研究结果显示，驱动忠实和不忠实错误还原的机制是不同的，这对 LLM 的推理是一个统一、连贯过程的观点提出了挑战。

语言模型中忠实与不忠实推理的分离

Dissociation of Faithful and Unfaithful Reasoning in LLMs

An explanation of a machine learning model is considered "faithful" if it
accurately reflects the model's decision-making process. However, explanations
such as feature attributions for deep learning are not guaranteed to be
faithful, and can produce potentially misleading interpretations. In this work,
we develop Sum-of-Parts (SOP), a class of models whose predictions come with
grouped feature attributions that are faithful-by-construction. This model
decomposes a prediction into an interpretable sum of scores, each of which is
directly attributable to a sparse group of features. We evaluate SOP on
benchmarks with standard interpretability metrics, and in a case study, we use
the faithful explanations from SOP to help astrophysicists discover new
knowledge about galaxy formation.

通过解释机器学习模型的决策过程来提供准确反映的解释被认为是 “忠实的”。本研究开发了一种名为 SOP 的模型类，其预测结果附带了忠实的分组特征归因。该模型将预测分解为可解释的得分总和，其中每个得分直接归因于一组稀疏特征。我们通过标准解释性度量和一个案例研究对 SOP 进行评估，并使用 SOP 提供的忠实解释帮助天体物理学家发现有关星系形成的新知识。