Recent studies show that instruction tuning and learning from human feedback
improve the abilities of large language models (LMs) dramatically. While these
tuning methods can make models generate high-quality text, we conjecture that
more implicit cognitive biases may arise in these fine-tuned models. Our work
provides evidence that these fine-tuned models exhibit biases that were absent
or less pronounced in their pretrained predecessors. We examine the extent of
this phenomenon in three cognitive biases - the decoy effect, the certainty
effect, and the belief bias - all of which are known to influence human
decision-making and reasoning. Our findings highlight the presence of these
biases in various models, especially those that have undergone instruction
tuning, such as Flan-T5, GPT3.5, and GPT4. This research constitutes a step
toward comprehending cognitive biases in instruction-tuned LMs, which is
crucial for the development of more reliable and unbiased language models.

通过检验三种认知偏见（假象效应、确定性效应和信仰偏误）在经过指导调优的语言模型中的存在程度，我们的研究提供了证据表明，这些经过调优的模型表现出过去预训练模型中不存在或较不明显的偏见，进一步突出了这些偏见存在于各种模型中的事实，特别是那些经过指导调优的模型，如 Flan-T5，GPT3.5 和 GPT4，这一研究对认识指导调优的语言模型中的认知偏见是至关重要的，从而对更可靠、无偏的语言模型的发展具有重要意义。

受指导的偏见：经过指导调节的语言模型呈现出应急认知偏差

Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent  Cognitive Bias

Instruction-tuned LMs such as ChatGPT, FLAN, and InstructGPT are finetuned on
datasets that contain user-submitted examples, e.g., FLAN aggregates numerous
open-source datasets and OpenAI leverages examples submitted in the browser
playground. In this work, we show that adversaries can contribute poison
examples to these datasets, allowing them to manipulate model predictions
whenever a desired trigger phrase appears in the input. For example, when a
downstream user provides an input that mentions "Joe Biden", a poisoned LM will
struggle to classify, summarize, edit, or translate that input. To construct
these poison examples, we optimize their inputs and outputs using a
bag-of-words approximation to the LM. We evaluate our method on open-source
instruction-tuned LMs. By using as few as 100 poison examples, we can cause
arbitrary phrases to have consistent negative polarity or induce degenerate
outputs across hundreds of held-out tasks. Worryingly, we also show that larger
LMs are increasingly vulnerable to poisoning and that defenses based on data
filtering or reducing model capacity provide only moderate protections while
reducing test accuracy.

本研究表明对 instruction-tuned LMs 输入恶意抽样将导致模型预测失准，大型 LMs 在此方面更易受攻击，而基于数据过滤或减少模型容量的防御措施提供的保护有限，同时会降低测试准确性。