Representation learning, and interpreting learned representations, are key
areas of focus in machine learning and neuroscience. Both fields generally use
representations as a means to understand or improve a system's computations. In
this work, however, we explore surprising dissociations between representation
and computation that may pose challenges for such efforts. We create datasets
in which we attempt to match the computational role that different features
play, while manipulating other properties of the features or the data. We train
various deep learning architectures to compute these multiple abstract features
about their inputs. We find that their learned feature representations are
systematically biased towards representing some features more strongly than
others, depending upon extraneous properties such as feature complexity, the
order in which features are learned, and the distribution of features over the
inputs. For example, features that are simpler to compute or learned first tend
to be represented more strongly and densely than features that are more complex
or learned later, even if all features are learned equally well. We also
explore how these biases are affected by architectures, optimizers, and
training regimes (e.g., in transformers, features decoded earlier in the output
sequence also tend to be represented more strongly). Our results help to
characterize the inductive biases of gradient-based representation learning.
These results also highlight a key challenge for interpretability $-$ or for
comparing the representations of models and brains $-$ disentangling extraneous
biases from the computationally important aspects of a system's internal
representations.

在这项工作中，我们探讨了表示学习和解释所学的表示在机器学习和神经科学中的关键领域，并发现了表示与计算之间一些令人惊讶的脱节现象，这可能对相关工作构成挑战。我们通过创建数据集来匹配不同特征的计算角色，并操纵特征或数据的其他属性。我们训练各种深度学习架构以计算这些关于输入的多个抽象特征，并发现他们学到的特征表示在表示一些特征时存在系统性的偏倚，这些偏倚依赖于特征的复杂性、特征学习的顺序和特征在输入中的分布。同时我们还探索了这些偏差对体系结构、优化器和训练方案的影响。这些结果有助于描述基于梯度的表示学习的归纳偏差，同时突出了理解性或对比模型和大脑之间表示的关键挑战，即区分系统内部表示的外部偏差和计算上重要的方面。