Attention mechanism has been widely believed as the key to success of vision
transformers (ViTs), since it provides a flexible and powerful way to model
spatial relationships. However, is the attention mechanism truly an
indispensable part of ViT? Can it be replaced by some other alternatives? To
demystify the role of attention mechanism, we simplify it into an extremely
simple case: ZERO FLOP and ZERO parameter. Concretely, we revisit the shift
operation. It does not contain any parameter or arithmetic calculation. The
only operation is to exchange a small portion of the channels between
neighboring features. Based on this simple operation, we construct a new
backbone network, namely ShiftViT, where the attention layers in ViT are
substituted by shift operations. Surprisingly, ShiftViT works quite well in
several mainstream tasks, e.g., classification, detection, and segmentation.
The performance is on par with or even better than the strong baseline Swin
Transformer. These results suggest that the attention mechanism might not be
the vital factor that makes ViT successful. It can be even replaced by a
zero-parameter operation. We should pay more attentions to the remaining parts
of ViT in the future work. Code is available at github.com/microsoft/SPACH.

本研究研究了注意力机制在视觉 Transformer 中的作用，将其简化为一个名为 ShiftViT 的零参数操作，发现它在分类、检测和分割等任务中表现良好，并且性能与强基线 Swin Transformer 相当甚至更好。

当位移操作遇上视觉 Transformer：一种极其简单的替代注意力机制的方法

When Shift Operation Meets Vision Transformer: An Extremely Simple  Alternative to Attention Mechanism

Modern single image super-resolution (SISR) system based on convolutional
neural networks (CNNs) achieves fancy performance while requires huge
computational costs. The problem on feature redundancy is well studied in
visual recognition task, but rarely discussed in SISR. Based on the observation
that many features in SISR models are also similar to each other, we propose to
use shift operation to generate the redundant features (i.e., ghost features).
Compared with depth-wise convolution which is time-consuming on GPU-like
devices, shift operation can bring a practical inference acceleration for CNNs
on common hardwares. We analyze the benefits of shift operation on SISR task
and make the shift orientation learnable based on Gumbel-Softmax trick.
Besides, a clustering procedure is explored based on pre-trained models to
identify the intrinsic filters for generating intrinsic features. The ghost
features will be derived by moving these intrinsic features along a specific
orientation. Finally, the complete output features are constructed by
concatenating the intrinsic and ghost features together. Extensive experiments
on several benchmark models and datasets demonstrate that both the non-compact
and lightweight SISR models embedded with the proposed method can achieve a
comparable performance to that of their baselines with a large reduction of
parameters, FLOPs and GPU inference latency. For instance, we reduce the
parameters by 46%, FLOPs by 46% and GPU inference latency by 42% of $\times2$
EDSR network with basically lossless performance.

本论文提出利用移位操作来生成冗余特征并构造完整特征，在多个基准模型和数据集上进行了广泛实验，结果显示所提出的方法嵌入了非紧凑和轻量级 SISR 模型，与基线模型相比，在大幅减少参数、FLOPs 和 GPU 推理延迟的同时，性能基本不变。

GhostSR: 学习幽灵特征以实现高效图像超分辨率

GhostSR: Learning Ghost Features for Efficient Image Super-Resolution

The shift operation was recently introduced as an alternative to spatial
convolutions. The operation moves subsets of activations horizontally and/or
vertically. Spatial convolutions are then replaced with shift operations
followed by point-wise convolutions, significantly reducing computational
costs. In this work, we investigate how shifts should best be applied to high
accuracy CNNs. We apply shifts of two different neighbourhood groups to ResNet
on ImageNet: the originally introduced 8-connected (8C) neighbourhood shift and
the less well studied 4-connected (4C) neighbourhood shift. We find that when
replacing ResNet's spatial convolutions with shifts, both shift neighbourhoods
give equal ImageNet accuracy, showing the sufficiency of small neighbourhoods
for large images. Interestingly, when incorporating shifts to all point-wise
convolutions in residual networks, 4-connected shifts outperform 8-connected
shifts. Such a 4-connected shift setup gives the same accuracy as full residual
networks while reducing the number of parameters and FLOPs by over 40%. We then
highlight that without spatial convolutions, ResNet's downsampling/upsampling
bottleneck channel structure is no longer needed. We show a new, 4C shift-based
residual network, much shorter than the original ResNet yet with a higher
accuracy for the same computational cost. This network is the highest accuracy
shift-based network yet shown, demonstrating the potential of shifting in deep
neural networks.

本文研究了如何应用转移运算以替代空间卷积操作，通过将 8-connected 和 4-connected 邻域组的转移操作应用于 ResNet 的 point-wise 卷积中，实现了与全残差网络相同的准确性，同时将参数和计算要求降低了 40％，通过减少横向连接层数来生成了一个准确性更高的新 ResNet 网络。

4 连通移位残差网络

4-Connected Shift Residual Networks

Neural networks rely on convolutions to aggregate spatial information.
However, spatial convolutions are expensive in terms of model size and
computation, both of which grow quadratically with respect to kernel size. In
this paper, we present a parameter-free, FLOP-free "shift" operation as an
alternative to spatial convolutions. We fuse shifts and point-wise convolutions
to construct end-to-end trainable shift-based modules, with a hyperparameter
characterizing the tradeoff between accuracy and efficiency. To demonstrate the
operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based
modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters;
we additionally demonstrate the operation's resilience to parameter reduction
on ImageNet, outperforming ResNet family members. We finally show the shift
operation's applicability across domains, achieving strong performance with
fewer parameters on classification, face verification and style transfer.

本文提出一种基于 shift 的操作，用于代替 spatial convolutions 提高神经网络的效率，并在分类、人脸识别、风格迁移等各类领域中取得了不错的表现。