Recent applications of LLMs in Machine Reading Comprehension (MRC) systems
have shown impressive results, but the use of shortcuts, mechanisms triggered
by features spuriously correlated to the true label, has emerged as a potential
threat to their reliability. We analyze the problem from two angles: LLMs as
editors, guided to edit text to mislead LLMs; and LLMs as readers, who answer
questions based on the edited text. We introduce a framework that guides an
editor to add potential shortcuts-triggers to samples. Using GPT4 as the
editor, we find it can successfully edit trigger shortcut in samples that fool
LLMs. Analysing LLMs as readers, we observe that even capable LLMs can be
deceived using shortcut knowledge. Strikingly, we discover that GPT4 can be
deceived by its own edits (15% drop in F1). Our findings highlight inherent
vulnerabilities of LLMs to shortcut manipulations. We publish ShortcutQA, a
curated dataset generated by our framework for future research.

近期应用于机器阅读理解（MRC）系统的 LLMs 显示出了令人印象深刻的结果，但使用的快捷方式，即特征与真实标签表面相关的机制，已成为其可靠性的潜在威胁。我们从两个角度分析了问题：LLMs 作为编辑器，被引导编辑文本以误导 LLMs；LLMs 作为读者，根据编辑后的文本回答问题。我们引入了一个框架，指导编辑器在样本中添加可能的快捷方式触发器。使用 GPT4 作为编辑器，我们发现它能够成功编辑样本中欺骗 LLMs 的快捷方式触发器。分析 LLMs 作为读者，我们观察到即使是能力强大的 LLMs 也可以被快捷方式知识欺骗。令人惊讶的是，我们发现 GPT4 甚至被自己的编辑欺骗（F1 下降了 15%）。我们的发现凸显了 LLMs 对于快捷方式操作的固有脆弱性。我们发布了 ShortcutQA，这是一个由我们的框架生成的精选数据集，用于未来研究。