We revisit and extend model stitching (Lenc & Vedaldi 2015) as a methodology
to study the internal representations of neural networks. Given two trained and
frozen models $A$ and $B$, we consider a "stitched model'' formed by connecting
the bottom-layers of $A$ to the top-layers of $B$, with a simple trainable
layer between them. We argue that model stitching is a powerful and perhaps
under-appreciated tool, which reveals aspects of representations that measures
such as centered kernel alignment (CKA) cannot. Through extensive experiments,
we use model stitching to obtain quantitative verifications for intuitive
statements such as "good networks learn similar representations'', by
demonstrating that good networks of the same architecture, but trained in very
different ways (e.g.: supervised vs. self-supervised learning), can be stitched
to each other without drop in performance. We also give evidence for the
intuition that "more is better'' by showing that representations learnt with
(1) more data, (2) bigger width, or (3) more training time can be "plugged in''
to weaker models to improve performance. Finally, our experiments reveal a new
structural property of SGD which we call "stitching connectivity'', akin to
mode-connectivity: typical minima reached by SGD can all be stitched to each
other with minimal change in accuracy.

本文重新审视和扩展了模型拼接的方法，通过将训练好的两个模型 A 和 B 底层连接到顶层，并在它们之间加入一个简单的可训练层来形成一个 “拼接模型”，揭示了多方面的表示属性。通过大量实验证明，“好” 的网络学习相似的表示，更多的数据，更大的宽度或更多的训练时间可以 “插入” 到较弱的模型中以提高性能。此外，我们还展示了 SGD 的一个新的结构特性，称为 “缝合连接”，类似于模式连接，SGD 达到的典型最小值可以通过最小的精度变化缝合在一起。