Neural networks exhibit simplicity bias; they rely on simpler features while
ignoring equally predictive but more complex features. In this work, we
introduce a novel approach termed imbalanced label coupling to investigate
scenarios where simple and complex features exhibit different levels of
predictive power. In these cases, complex features still contribute to
predictions. The trained networks make predictions in alignment with the
ascending complexity of input features according to how they correlate with the
label in the training set, irrespective of the underlying predictive power. For
instance, even when simple spurious features distort predictions in CIFAR-10,
most cats are predicted to be dogs, and most trucks are predicted to be
automobiles! This observation provides direct evidence that the neural network
learns core features in the presence of spurious features. We empirically show
that last-layer retraining with target data distribution is effective, yet
insufficient to fully recover core features when spurious features are
perfectly correlated with the target labels in our synthetic dataset. We hope
our research contributes to a deeper understanding of the implicit bias of
neural networks.

神经网络在存在偏执特征时，能够学习到核心特征，即使简单虚假特征扭曲了预测结果；研究通过不平衡标签耦合的方法，调查了简单和复杂特征在预测中的不同水平，以及复杂特征在预测中的贡献。