Recent isotropic networks, such as ConvMixer and vision transformers, have
found significant success across visual recognition tasks, matching or
outperforming non-isotropic convolutional neural networks (CNNs). Isotropic
architectures are particularly well-suited to cross-layer weight sharing, an
effective neural network compression technique. In this paper, we perform an
empirical evaluation on methods for sharing parameters in isotropic networks
(SPIN). We present a framework to formalize major weight sharing design
decisions and perform a comprehensive empirical evaluation of this design
space. Guided by our experimental results, we propose a weight sharing strategy
to generate a family of models with better overall efficiency, in terms of
FLOPs and parameters versus accuracy, compared to traditional scaling methods
alone, for example compressing ConvMixer by 1.9x while improving accuracy on
ImageNet. Finally, we perform a qualitative study to further understand the
behavior of weight sharing in isotropic architectures. The code is available at
this https URL

本研究旨在对等向网络 (SPIN) 进行经验性评价，提出了一种权值共享策略，并对权值共享的行为进行了定性研究。实验结果表明，该策略可以比传统缩放方法更有效地提高模型效率，例如在 ImageNet 数据集上将 ConvMixer 压缩 1.9x 并提高准确性

一个关于等向网络参数共享的实证评估

SPIN: An Empirical Evaluation on Sharing Parameters of Isotropic  Networks

Although convolutional networks have been the dominant architecture for
vision tasks for many years, recent experiments have shown that
Transformer-based models, most notably the Vision Transformer (ViT), may exceed
their performance in some settings. However, due to the quadratic runtime of
the self-attention layers in Transformers, ViTs require the use of patch
embeddings, which group together small regions of the image into single input
features, in order to be applied to larger image sizes. This raises a question:
Is the performance of ViTs due to the inherently-more-powerful Transformer
architecture, or is it at least partly due to using patches as the input
representation? In this paper, we present some evidence for the latter:
specifically, we propose the ConvMixer, an extremely simple model that is
similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it
operates directly on patches as input, separates the mixing of spatial and
channel dimensions, and maintains equal size and resolution throughout the
network. In contrast, however, the ConvMixer uses only standard convolutions to
achieve the mixing steps. Despite its simplicity, we show that the ConvMixer
outperforms the ViT, MLP-Mixer, and some of their variants for similar
parameter counts and data set sizes, in addition to outperforming classical
vision models such as the ResNet. Our code is available at
this https URL.

本文介绍 ConvMixer 模型，它使用标准卷积来混合图像块，并在类似参数计数和数据集大小的情况下胜过 ViT、MLP-Mixer 和一些变种，同时也优于经典的 ResNet 等视觉模型。