In recent years, Transformer networks are beginning to replace pure
convolutional neural networks (CNNs) in the field of computer vision due to
their global receptive field and adaptability to input. However, the quadratic
computational complexity of softmax-attention limits the wide application in
image dehazing task, especially for high-resolution images. To address this
issue, we propose a new Transformer variant, which applies the Taylor expansion
to approximate the softmax-attention and achieves linear computational
complexity. A multi-scale attention refinement module is proposed as a
complement to correct the error of the Taylor expansion. Furthermore, we
introduce a multi-branch architecture with multi-scale patch embedding to the
proposed Transformer, which embeds features by overlapping deformable
convolution of different scales. The design of multi-scale patch embedding is
based on three key ideas: 1) various sizes of the receptive field; 2)
multi-level semantic information; 3) flexible shapes of the receptive field.
Our model, named Multi-branch Transformer expanded by Taylor formula
(MB-TaylorFormer), can embed coarse to fine features more flexibly at the patch
embedding stage and capture long-distance pixel interactions with limited
computational cost. Experimental results on several dehazing benchmarks show
that MB-TaylorFormer achieves state-of-the-art (SOTA) performance with a light
computational burden. The source code and pre-trained models are available at
this https URL

近年来，Transformer 网络开始取代纯卷积神经网络在计算机视觉领域的应用，但是其二次计算复杂度的 softmax-attention 限制了高分辨率图像去雾任务的广泛应用。因此，我们提出了一种新的 Transformer 变体，应用泰勒展开逼近 softmax-attention，实现了线性计算复杂度，并引入了多尺度注意力细化模块和多分支架构以及多尺度补丁嵌入方法进一步改进 Transformer 的性能。我们的模型名为 MB-TaylorFormer，在补丁嵌入阶段可以更灵活地嵌入粗糙到细粒度的特征，并以有限的计算成本捕捉远距离像素之间的相互作用。实验证明，MB-TaylorFormer 在多个去雾基准上取得了最先进的性能，并具有轻量级的计算负担。源代码和预训练模型可在此 https URL 获得。

MB-TaylorFormer：基于 Taylor 公式扩展的多分支高效 Transformer 用于图像去雾

MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor  Formula for Image Dehazing

Despite the recent success in many applications, the high computational
requirements of vision transformers limit their use in resource-constrained
settings. While many existing methods improve the quadratic complexity of
attention, in most vision transformers, self-attention is not the major
computation bottleneck, e.g., more than 80% of the computation is spent on
fully-connected layers. To improve the computational complexity of all layers,
we propose a novel token downsampling method, called Token Pooling, efficiently
exploiting redundancies in the images and intermediate token representations.
We show that, under mild assumptions, softmax-attention acts as a
high-dimensional low-pass (smoothing) filter. Thus, its output contains
redundancy that can be pruned to achieve a better trade-off between the
computational cost and accuracy. Our new technique accurately approximates a
set of tokens by minimizing the reconstruction error caused by downsampling. We
solve this optimization problem via cost-efficient clustering. We rigorously
analyze and compare to prior downsampling methods. Our experiments show that
Token Pooling significantly improves the cost-accuracy trade-off over the
state-of-the-art downsampling. Token Pooling is a simple and effective operator
that can benefit many architectures. Applied to DeiT, it achieves the same
ImageNet top-1 accuracy using 42% fewer computations.

该研究提出了一种称作 Token Pooling 的新型令牌下采样方法，旨在提高对视觉变换的计算速度，并通过对 softmax 注意力机制的研究，实现了更好的计算速度与精度之间的平衡。

视觉 Transformer 中的 Token 池化

Token Pooling in Vision Transformers

Attention architectures are widely used; they recently gained renewed
popularity with Transformers yielding a streak of state of the art results.
Yet, the geometrical implications of softmax-attention remain largely
unexplored. In this work we highlight the limitations of constraining attention
weights to the probability simplex and the resulting convex hull of value
vectors. We show that Transformers are sequence length dependent biased towards
token isolation at initialization and contrast Transformers to simple max- and
sum-pooling - two strong baselines rarely reported. We propose to replace the
softmax in self-attention with normalization, yielding a hyperparameter and
data-bias robust, generally applicable architecture. We support our insights
with empirical results from more than 25,000 trained models. All results and
implementations are made available.

本文介绍了 softmax-attention 在几何上的局限性，并提出采用归一化代替 softmax 实现自我注意力，从而获得超参数和数据推断鲁棒性较强的通用结构。