Though U-Net has achieved tremendous success in medical image segmentation
tasks, it lacks the ability to explicitly model long-range dependencies.
Therefore, Vision Transformers have emerged as alternative segmentation
structures recently, for their innate ability of capturing long-range
correlations through Self-Attention (SA). However, Transformers usually rely on
large-scale pre-training and have high computational complexity. Furthermore,
SA can only model self-affinities within a single sample, ignoring the
potential correlations of the overall dataset. To address these problems, we
propose a novel Transformer module named Mixed Transformer Module (MTM) for
simultaneous inter- and intra- affinities learning. MTM first calculates
self-affinities efficiently through our well-designed Local-Global
Gaussian-Weighted Self-Attention (LGG-SA). Then, it mines inter-connections
between data samples through External Attention (EA). By using MTM, we
construct a U-shaped model named Mixed Transformer U-Net (MT-UNet) for accurate
medical image segmentation. We test our method on two different public
datasets, and the experimental results show that the proposed method achieves
better performance over other state-of-the-art methods. The code is available
at: this https URL

本研究提出了一种新颖的混合变压器模块（MTM），用于同时学习内部和外部联系，通过使用 MTM 构建的混合变压器 U-Net（MT-UNet）来进行精确的医学图像分割，实验结果显示，该方法实现了比其他最新技术方法更好的性能。

混合 Transformer U-Net 用于医学图像分割

Mixed Transformer U-Net For Medical Image Segmentation

Attention mechanisms, especially self-attention, have played an increasingly
important role in deep feature representation for visual tasks. Self-attention
updates the feature at each position by computing a weighted sum of features
using pair-wise affinities across all positions to capture the long-range
dependency within a single sample. However, self-attention has quadratic
complexity and ignores potential correlation between different samples. This
paper proposes a novel attention mechanism which we call external attention,
based on two external, small, learnable, shared memories, which can be
implemented easily by simply using two cascaded linear layers and two
normalization layers; it conveniently replaces self-attention in existing
popular architectures. External attention has linear complexity and implicitly
considers the correlations between all data samples. We further incorporate the
multi-head mechanism into external attention to provide an all-MLP
architecture, external attention MLP (EAMLP), for image classification.
Extensive experiments on image classification, object detection, semantic
segmentation, instance segmentation, image generation, and point cloud analysis
reveal that our method provides results comparable or superior to the
self-attention mechanism and some of its variants, with much lower
computational and memory costs.

本文提出了外部注意力机制，它在现有流行的架构中方便地替换自我注意力，具有线性复杂度，隐含地考虑所有数据样本之间的关联，结合多头机制提供了外部注意力 MLP（EAMLP）体系结构，用于图像分类和其他任务中可提供与或优于自我注意力机制及其变体相当的结果，且计算和内存成本较低。

超越自注意力：用两个线性层的外部注意力处理视觉任务

Beyond Self-attention: External Attention using Two Linear Layers for  Visual Tasks

Neural networks with attention have proven effective for many natural
language processing tasks. In this paper, we develop attention mechanisms for
uncertainty detection. In particular, we generalize standardly used attention
mechanisms by introducing external attention and sequence-preserving attention.
These novel architectures differ from standard approaches in that they use
external resources to compute attention weights and preserve sequence
information. We compare them to other configurations along different dimensions
of attention. Our novel architectures set the new state of the art on a
Wikipedia benchmark dataset and perform similar to the state-of-the-art model
on a biomedical benchmark which uses a large set of linguistic features.

本文中我们为不确定性检测开发了注意力机制的神经网络，引入外部注意力和序列保持注意力等新的架构，并通过多个注意力维度与其他配置进行了比较。这些新的架构在一个 Wikipedia 基准数据集上取得了新的 state-of-the-art，在使用大量语言特征的生物医学基准测试中与现有技术相似。