Non-local (NL) block is a popular module that demonstrates the capability to
model global contexts. However, NL block generally has heavy computation and
memory costs, so it is impractical to apply the block to high-resolution
feature maps. In this paper, to investigate the efficacy of NL block, we
empirically analyze if the magnitude and direction of input feature vectors
properly affect the attention between vectors. The results show the inefficacy
of softmax operation which is generally used to normalize the attention map of
the NL block. Attention maps normalized with softmax operation highly rely upon
magnitude of key vectors, and performance is degenerated if the magnitude
information is removed. By replacing softmax operation with the scaling factor,
we demonstrate improved performance on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
In Addition, our method shows robustness to embedding channel reduction and
embedding weight initialization. Notably, our method makes multi-head attention
employable without additional computational cost.

本文通过实验证明 NL block 中 softmax 操作效率低下，提出使用 scaling factor 来优化 attention maps，取得了在 CIFAR-10、CIFAR-100 和 Tiny-ImageNet 数据集上的改进，并且在多头注意力下没有额外的计算成本。

重新思考轻量级非局部神经网络中 Softmax 的有效性

Rethinking Efficacy of Softmax for Lightweight Non-Local Neural Networks

The non-local block is a popular module for strengthening the context
modeling ability of a regular convolutional neural network. This paper first
studies the non-local block in depth, where we find that its attention
computation can be split into two terms, a whitened pairwise term accounting
for the relationship between two pixels and a unary term representing the
saliency of every pixel. We also observe that the two terms trained alone tend
to model different visual clues, e.g. the whitened pairwise term learns
within-region relationships while the unary term learns salient boundaries.
However, the two terms are tightly coupled in the non-local block, which
hinders the learning of each. Based on these findings, we present the
disentangled non-local block, where the two terms are decoupled to facilitate
learning for both terms. We demonstrate the effectiveness of the decoupled
design on various tasks, such as semantic segmentation on Cityscapes, ADE20K
and PASCAL Context, object detection on COCO, and action recognition on
Kinetics.

本文研究了非局部块，发现其关注计算可以分为两个项，其中白化的成对项和一元项分别代表两个像素之间的关系及每个像素的显著性。本文提出了解耦的非局部块，证明了该设计在不同任务中的有效性。

非局部解耦神经网络

Disentangled Non-Local Neural Networks

Video-based person re-identification (ReID) is a challenging problem, where
some video tracks of people across non-overlapping cameras are available for
matching. Feature aggregation from a video track is a key step for video-based
person ReID. Many existing methods tackle this problem by average/maximum
temporal pooling or RNNs with attention. However, these methods cannot deal
with temporal dependency and spatial misalignment problems at the same time. We
are inspired by video action recognition that involves the identification of
different actions from video tracks. Firstly, we use 3D convolutions on video
volume, instead of using 2D convolutions across frames, to extract spatial and
temporal features simultaneously. Secondly, we use a non-local block to tackle
the misalignment problem and capture spatial-temporal long-range dependencies.
As a result, the network can learn useful spatial-temporal information as a
weighted sum of the features in all space and temporal positions in the input
feature map. Experimental results on three datasets show that our framework
outperforms state-of-the-art approaches by a large margin on multiple metrics.

本研究旨在解决视频行人重新识别中的空间对齐问题和时间依赖性问题，提出了一种使用 3D 卷积和非局部块进行特征聚合的网络架构，实验结果表明该方法在多个指标上的性能均优于现有技术。