There has been increasing interest in evaluations of language models for a
variety of risks and characteristics. Evaluations relying on natural language
understanding for grading can often be performed at scale by using other
language models. We test the robustness of these model-graded evaluations to
injections on different datasets including a new Deception Eval. These
injections resemble direct communication between the testee and the evaluator
to change their grading. We extrapolate that future, more intelligent models
might manipulate or cooperate with their evaluation model. We find significant
susceptibility to these injections in state-of-the-art commercial models on all
examined evaluations. Furthermore, similar injections can be used on automated
interpretability frameworks to produce misleading model-written explanations.
The results inspire future work and should caution against unqualified trust in
evaluations and automated interpretability.

通过对语言模型进行评估，我们发现其在不同数据集中，包括新的虚假评估中，对评估注入的改变非常敏感。类似的注入也可用于自动的解释性框架，以产生误导性的模型编写解释。这些结果激发了进一步的研究，并应该警告我们不要对评估和自动解释过度信任。