Despite their impressive performance in object recognition and other tasks
under standard testing conditions, deep networks often fail to generalize to
out-of-distribution (o.o.d.) samples. One cause for this shortcoming is that
modern architectures tend to rely on "shortcuts" - superficial features that
correlate with categories without capturing deeper invariants that hold across
contexts. Real-world concepts often possess a complex structure that can vary
superficially across contexts, which can make the most intuitive and promising
solutions in one context not generalize to others. One potential way to improve
o.o.d. generalization is to assume simple solutions are unlikely to be valid
across contexts and avoid them, which we refer to as the too-good-to-be-true
prior. A low-capacity network (LCN) with a shallow architecture should only be
able to learn surface relationships, including shortcuts. We find that LCNs can
serve as shortcut detectors. Furthermore, an LCN's predictions can be used in a
two-stage approach to encourage a high-capacity network (HCN) to rely on deeper
invariant features that should generalize broadly. In particular, items that
the LCN can master are downweighted when training the HCN. Using a modified
version of the CIFAR-10 dataset in which we introduced shortcuts, we found that
the two-stage LCN-HCN approach reduced reliance on shortcuts and facilitated
o.o.d. generalization.

研究了深度网络在 o.o.d. 样本下的推广问题，提出了使用浅层结构的低容量网络侦测卷积神经网络中的 superficial features，从而避免过度依赖 “捷径” 解决问题，并鼓励深层次学习。实验证明，这种两阶段的 LCN-HCN 方法有助于减少对捷径的依赖和促进 o.o.d. 推广。