This paper studies how to keep a vision backbone effective while removing
token mixers in its basic building blocks. Token mixers, as self-attention for
vision transformers (ViTs), are intended to perform information communication
between different spatial tokens but suffer from considerable computational
cost and latency. However, directly removing them will lead to an incomplete
model structure prior, and thus brings a significant accuracy drop. To this
end, we first develop an RepIdentityFormer base on the re-parameterizing idea,
to study the token mixer free model architecture. And we then explore the
improved learning paradigm to break the limitation of simple token mixer free
backbone, and summarize the empirical practice into 5 guidelines. Equipped with
the proposed optimization strategy, we are able to build an extremely simple
vision backbone with encouraging performance, while enjoying the high
efficiency during inference. Extensive experiments and ablative analysis also
demonstrate that the inductive bias of network architecture, can be
incorporated into simple network structure with appropriate optimization
strategy. We hope this work can serve as a starting point for the exploration
of optimization-driven efficient network design. Project page:
this https URL

本文研究如何在去除基本构建模块中的令牌混合器的同时保持视觉骨干的有效性，并提出了一个可行的优化策略，使得我们能够构建一种极其简单的视觉骨干，具有鼓舞人心的性能，同时在推理过程中享受高效性。

RIFormer：在不使用 Token Mixer 功能的同时保持视觉骨干网络的有效性

RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer

Previous feed-forward architectures of recently proposed deep
super-resolution networks learn the features of low-resolution inputs and the
non-linear mapping from those to a high-resolution output. However, this
approach does not fully address the mutual dependencies of low- and
high-resolution images. We propose Deep Back-Projection Networks (DBPN), the
winner of two image super-resolution challenges (NTIRE2018 and PIRM2018), that
exploit iterative up- and down-sampling layers. These layers are formed as a
unit providing an error feedback mechanism for projection errors. We construct
mutually-connected up- and down-sampling units each of which represents
different types of low- and high-resolution components. We also show that
extending this idea to demonstrate a new insight towards more efficient network
design substantially, such as parameter sharing on the projection module and
transition layer on projection step. The experimental results yield superior
results and in particular establishing new state-of-the-art results across
multiple data sets, especially for large scaling factors such as 8x.

Deep Back-Projection Networks (DBPN) effectively address the mutual dependencies of low- and high-resolution images through iterative up- and down-sampling layers and mutual connections, achieving state-of-the-art results in image super-resolution with large scaling factors such as 8x.