We study neural network loss landscapes through the lens of mode
connectivity, the observation that minimizers of neural networks retrieved via
training on a dataset are connected via simple paths of low loss. Specifically,
we ask the following question: are minimizers that rely on different mechanisms
for making their predictions connected via simple paths of low loss? We provide
a definition of mechanistic similarity as shared invariances to input
transformations and demonstrate that lack of linear connectivity between two
models implies they use dissimilar mechanisms for making their predictions.
Relevant to practice, this result helps us demonstrate that naive fine-tuning
on a downstream dataset can fail to alter a model's mechanisms, e.g.,
fine-tuning can fail to eliminate a model's reliance on spurious attributes.
Our analysis also motivates a method for targeted alteration of a model's
mechanisms, named connectivity-based fine-tuning (CBFT), which we analyze using
several synthetic datasets for the task of reducing a model's reliance on
spurious attributes.

本文研究神经网络损失景观的模式连接性，提出了机制相似性的定义，并证明缺乏线性连接意味着两个模型使用不同的机制进行预测。作者还介绍了一种名为基于连接性的微调方法用于正确修改一个模型的机制，以减少对于虚假属性的依赖。