Common explanations for shortcut learning assume that the shortcut improves
prediction under the training distribution but not in the test distribution.
Thus, models trained via the typical gradient-based optimization of
cross-entropy, which we call default-ERM, utilize the shortcut. However, even
when the stable feature determines the label in the training distribution and
the shortcut does not provide any additional information, like in perception
tasks, default-ERM still exhibits shortcut learning. Why are such solutions
preferred when the loss for default-ERM can be driven to zero using the stable
feature alone? By studying a linear perception task, we show that default-ERM's
preference for maximizing the margin leads to models that depend more on the
shortcut than the stable feature, even without overparameterization. This
insight suggests that default-ERM's implicit inductive bias towards max-margin
is unsuitable for perception tasks. Instead, we develop an inductive bias
toward uniform margins and show that this bias guarantees dependence only on
the perfect stable feature in the linear perception task. We develop loss
functions that encourage uniform-margin solutions, called margin control
(MARG-CTRL). MARG-CTRL mitigates shortcut learning on a variety of vision and
language tasks, showing that better inductive biases can remove the need for
expensive two-stage shortcut-mitigating methods in perception tasks.

对于感知任务而言，默认的 ERM 的隐含归纳偏差偏向于最大化边界学习模型，该偏差通过鼓励均匀边界解决方案可以消除 shortcut 学习，并在视觉和语言任务中证明更好的归纳偏差可以消除费时的两阶段 shortcut 缓解方法。

不要怪罪数据集偏移！梯度和交叉熵导致的快捷学习

Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross  Entropy

Recently Shah et al., 2020 pointed out the pitfalls of the simplicity bias -
the tendency of gradient-based algorithms to learn simple models - which
include the model's high sensitivity to small input perturbations, as well as
sub-optimal margins. In particular, while Stochastic Gradient Descent yields
max-margin boundary on linear models, such guarantee does not extend to
non-linear models. To mitigate the simplicity bias, we consider
uncertainty-driven perturbations (UDP) of the training data points, obtained
iteratively by following the direction that maximizes the model's estimated
uncertainty. The uncertainty estimate does not rely on the input's label and it
is highest at the decision boundary, and - unlike loss-driven perturbations -
it allows for using a larger range of values for the perturbation magnitude.
Furthermore, as real-world datasets have non-isotropic distances between data
points of different classes, the above property is particularly appealing for
increasing the margin of the decision boundary, which in turn improves the
model's generalization. We show that UDP is guaranteed to achieve the maximum
margin decision boundary on linear models and that it notably increases it on
challenging simulated datasets. For nonlinear models, we show empirically that
UDP reduces the simplicity bias and learns more exhaustive features.
Interestingly, it also achieves competitive loss-based robustness and
generalization trade-off on several datasets.

本文讨论了梯度下降算法中的简单性偏差问题，并提出一种基于不确定性驱动的扰动方式来减小这种偏差。我们发现该方法能够提高模型的边际及泛化性能，同时在多个数据集上表现出有竞争力的鲁棒性和泛化性的平衡。