Transformers have emerged as the state-of-the-art architecture in medical
image registration, outperforming convolutional neural networks (CNNs) by
addressing their limited receptive fields and overcoming gradient instability
in deeper models. Despite their success, transformer-based models require
substantial resources for training, including data, memory, and computational
power, which may restrict their applicability for end users with limited
resources. In particular, existing transformer-based 3D image registration
architectures face three critical gaps that challenge their efficiency and
effectiveness. Firstly, while mitigating the quadratic complexity of full
attention by focusing on local regions, window-based attention mechanisms often
fail to adequately integrate local and global information. Secondly, feature
similarities across attention heads that were recently found in multi-head
attention architectures indicate a significant computational redundancy,
suggesting that the capacity of the network could be better utilized to enhance
performance. Lastly, the granularity of tokenization, a key factor in
registration accuracy, presents a trade-off; smaller tokens improve detail
capture at the cost of higher computational complexity, increased memory
demands, and a risk of overfitting. Here, we propose EfficientMorph, a
transformer-based architecture for unsupervised 3D image registration. It
optimizes the balance between local and global attention through a plane-based
attention mechanism, reduces computational redundancy via cascaded group
attention, and captures fine details without compromising computational
efficiency, thanks to a Hi-Res tokenization strategy complemented by merging
operations. Notably, EfficientMorph sets a new benchmark for performance on the
OASIS dataset with 16-27x fewer parameters.

EfficientMorph 是一种基于 transformer 的无监督三维图像配准架构，通过平面注意机制优化本地和全局注意的平衡，通过级联组注意减少计算冗余，并通过 Hi-Res 令牌化策略捕捉细节，同时保持计算效率，在 OASIS 数据集上创下了 16-27 倍少参数的性能新纪录。

EfficientMorph：高参数效率的基于 Transformer 的三维图像配准架构

EfficientMorph: Parameter-Efficient Transformer-Based Architecture for  3D Image Registration

Many problems in computer vision require dealing with sparse, unordered data
in the form of point clouds. Permutation-equivariant networks have become a
popular solution-they operate on individual data points with simple perceptrons
and extract contextual information with global pooling. This can be achieved
with a simple normalization of the feature maps, a global operation that is
unaffected by the order. In this paper, we propose Attentive Context
Normalization (ACN), a simple yet effective technique to build
permutation-equivariant networks robust to outliers. Specifically, we show how
to normalize the feature maps with weights that are estimated within the
network, excluding outliers from this normalization. We use this mechanism to
leverage two types of attention: local and global-by combining them, our method
is able to find the essential data points in high-dimensional space to solve a
given task. We demonstrate through extensive experiments that our approach,
which we call Attentive Context Networks (ACNe), provides a significant leap in
performance compared to the state-of-the-art on camera pose estimation, robust
fitting, and point cloud classification under noise and outliers. Source code:
this https URL

本文提出了注意力上下文归一化（ACN）技术，基于排列不变网络实现离群点鲁棒性的全局归一化方法，并利用本地和全局注意力找到高维空间中的关键数据点来解决特定任务，实验证明我们的方法在处理稀疏、无序数据的点云分类、抗噪点云拟合和相机姿态估计等方面性能较现有方法显著提升。