Question answering (QA) models for reading comprehension tend to learn
shortcut solutions rather than the solutions intended by QA datasets. QA models
that have learned shortcut solutions can achieve human-level performance in
shortcut examples where shortcuts are valid, but these same behaviors degrade
generalization potential on anti-shortcut examples where shortcuts are invalid.
Various methods have been proposed to mitigate this problem, but they do not
fully take the characteristics of shortcuts themselves into account. We assume
that the learnability of shortcuts, i.e., how easy it is to learn a shortcut,
is useful to mitigate the problem. Thus, we first examine the learnability of
the representative shortcuts on extractive and multiple-choice QA datasets.
Behavioral tests using biased training sets reveal that shortcuts that exploit
answer positions and word-label correlations are preferentially learned for
extractive and multiple-choice QA, respectively. We find that the more
learnable a shortcut is, the flatter and deeper the loss landscape is around
the shortcut solution in the parameter space. We also find that the
availability of the preferred shortcuts tends to make the task easier to
perform from an information-theoretic viewpoint. Lastly, we experimentally show
that the learnability of shortcuts can be utilized to construct an effective QA
training set; the more learnable a shortcut is, the smaller the proportion of
anti-shortcut examples required to achieve comparable performance on shortcut
and anti-shortcut examples. We claim that the learnability of shortcuts should
be considered when designing mitigation methods.

研究发现，QA 模型学习步骤通常倾向于学习 shortcut solutions 而非 QA 数据集所期望的解题方法，而学习 shortcut solutions 的行为扰乱了 QA 模型在 anti-shortcut examples 上的泛化能力。研究人员提出利用 shortcut solutions 的可学习性来设计 QA 训练集，以降低 shortcut solutions 的学习程度，从而提高 QA 模型的准确性。

问答模型更喜欢学习哪种快速解决方案？

Which Shortcut Solution Do Question Answering Models Prefer to Learn?

Across applications spanning supervised classification and sequential
control, deep learning has been reported to find "shortcut" solutions that fail
catastrophically under minor changes in the data distribution. In this paper,
we show empirically that DNNs can be coaxed to avoid poor shortcuts by
providing an additional "priming" feature computed from key input features,
usually a coarse output estimate. Priming relies on approximate domain
knowledge of these task-relevant key input features, which is often easy to
obtain in practical settings. For example, one might prioritize recent frames
over past frames in a video input for visual imitation learning, or salient
foreground over background pixels for image classification. On NICO image
classification, MuJoCo continuous control, and CARLA autonomous driving, our
priming strategy works significantly better than several popular
state-of-the-art approaches for feature selection and data augmentation. We
connect these empirical findings to recent theoretical results on DNN
optimization, and argue theoretically that priming distracts the optimizer away
from poor shortcuts by creating better, simpler shortcuts.

本文通过实证分析表明新加基于先验知识的 priming 特征（例如通过视觉学习中的最近一帧或者图像分类中的醒目前景进行分类）可以显著提高深度学习对图像分类、持续控制和自主驾驶等应用的效果，从而较好地避免训练时出现的 “短路” 现象。理论上，这是因为 priming 特征引导了模型优化的方向，使其避免陷入一些不可靠的简单捷径。