Language models (LMs) have become important tools in a variety of
applications, from data processing to the creation of instruction-following
assistants. But despite their advantages, LMs have certain idiosyncratic
limitations such as the problem of `strong priors', where a model learns to
output typical continuations in response to certain, usually local, portions of
the input regardless of any earlier instructions. For example, prompt injection
attacks can induce models to ignore explicit directives. In some cases, larger
models have been shown to be more susceptible to these problems than similar
smaller models, an example of the phenomenon of `inverse scaling'. We develop a
new technique for mitigating the problem of strong priors: we take the original
set of instructions, produce a weakened version of the original prompt that is
even more susceptible to the strong priors problem, and then extrapolate the
continuation away from the weakened prompt. This lets us infer how the model
would continue a hypothetical strengthened set of instructions. Our technique
conceptualises LMs as mixture models which combine a family of data generation
processes, reinforcing the desired elements of the mixture. Our approach works
at inference time, removing any need for retraining. We apply it to eleven
models including GPT-2, GPT-3, Llama 2, and Mistral on four tasks, and find
improvements in 41/44. Across all 44 combinations the median increase in
proportion of tasks completed is 40%.

我们开发了一种新技术来减轻强偏好问题，通过将原始指令集产生一个弱化版本的提示并从弱化的提示中推断模型如何继续执行一个假设加强的指令集，我们将语言模型概念化为混合模型，并对 GPT-2、GPT-3、Llama 2 和 Mistral 等模型进行了应用，在四个任务中找到了 41/44 的改进，在所有 44 个组合中，完成任务比例的中值增加了 40%。