Deep learning models often rely only on a small set of features even when
there is a rich set of predictive signals in the training data. This makes
models brittle and sensitive to distribution shifts. In this work, we first
examine vision transformers (ViTs) and find that they tend to extract robust
and spurious features with distinct attention heads. As a result of this
modularity, their performance under distribution shifts can be significantly
improved at test time by pruning heads corresponding to spurious features,
which we demonstrate using an "oracle selection" on validation data. Second, we
propose a method to further enhance the diversity and complementarity of the
learned features by encouraging orthogonality of the attention heads' input
gradients. We observe improved out-of-distribution performance on diagnostic
benchmarks (MNIST-CIFAR, Waterbirds) as a consequence of the enhanced diversity
of features and the pruning of undesirable heads.

通过修剪与不必要特征相关的注意力头和鼓励输入梯度正交性，我们提出一种方法提高深度学习模型对分布变化的适应性并增加特征的多样性和互补性。