This paper studies how to keep a vision backbone effective while removing
token mixers in its basic building blocks. Token mixers, as self-attention for
vision transformers (ViTs), are intended to perform information communication
between different spatial tokens but suffer from considerable computational
cost and latency. However, directly removing them will lead to an incomplete
model structure prior, and thus brings a significant accuracy drop. To this
end, we first develop an RepIdentityFormer base on the re-parameterizing idea,
to study the token mixer free model architecture. And we then explore the
improved learning paradigm to break the limitation of simple token mixer free
backbone, and summarize the empirical practice into 5 guidelines. Equipped with
the proposed optimization strategy, we are able to build an extremely simple
vision backbone with encouraging performance, while enjoying the high
efficiency during inference. Extensive experiments and ablative analysis also
demonstrate that the inductive bias of network architecture, can be
incorporated into simple network structure with appropriate optimization
strategy. We hope this work can serve as a starting point for the exploration
of optimization-driven efficient network design. Project page:
this https URL

本文研究如何在去除基本构建模块中的令牌混合器的同时保持视觉骨干的有效性，并提出了一个可行的优化策略，使得我们能够构建一种极其简单的视觉骨干，具有鼓舞人心的性能，同时在推理过程中享受高效性。