Non-local (NL) block is a popular module that demonstrates the capability to
model global contexts. However, NL block generally has heavy computation and
memory costs, so it is impractical to apply the block to high-resolution
feature maps. In this paper, to investigate the efficacy of NL block, we
empirically analyze if the magnitude and direction of input feature vectors
properly affect the attention between vectors. The results show the inefficacy
of softmax operation which is generally used to normalize the attention map of
the NL block. Attention maps normalized with softmax operation highly rely upon
magnitude of key vectors, and performance is degenerated if the magnitude
information is removed. By replacing softmax operation with the scaling factor,
we demonstrate improved performance on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
In Addition, our method shows robustness to embedding channel reduction and
embedding weight initialization. Notably, our method makes multi-head attention
employable without additional computational cost.

本文通过实验证明 NL block 中 softmax 操作效率低下，提出使用 scaling factor 来优化 attention maps，取得了在 CIFAR-10、CIFAR-100 和 Tiny-ImageNet 数据集上的改进，并且在多头注意力下没有额外的计算成本。

重新思考轻量级非局部神经网络中 Softmax 的有效性

Rethinking Efficacy of Softmax for Lightweight Non-Local Neural Networks

Although autoregressive models have achieved promising results on image
generation, their unidirectional generation process prevents the resultant
images from fully reflecting global contexts. To address the issue, we propose
an effective image generation framework of Draft-and-Revise with Contextual
RQ-transformer to consider global contexts during the generation process. As a
generalized VQ-VAE, RQ-VAE first represents a high-resolution image as a
sequence of discrete code stacks. After code stacks in the sequence are
randomly masked, Contextual RQ-Transformer is trained to infill the masked code
stacks based on the unmasked contexts of the image. Then, Contextual
RQ-Transformer uses our two-phase decoding, Draft-and-Revise, and generates an
image, while exploiting the global contexts of the image during the generation
process. Specifically. in the draft phase, our model first focuses on
generating diverse images despite rather low quality. Then, in the revise
phase, the model iteratively improves the quality of images, while preserving
the global contexts of generated images. In experiments, our method achieves
state-of-the-art results on conditional image generation. We also validate that
the Draft-and-Revise decoding can achieve high performance by effectively
controlling the quality-diversity trade-off in image generation.

该研究提出了一种有效的基于文本上下文和全局语境的图像生成框架，名为 Draft-and-Revise with Contextual RQ-transformer，通过 Contextual RQ-transformer 考虑图像全局上下文信息以最大程度地反映全局语境。在实验中，我们的方法在条件图像生成上达到了最先进的效果。

草稿与修订：基于上下文 RQ-Transformer 的有效图像生成

Draft-and-Revise: Effective Image Generation with Contextual  RQ-Transformer

Medical image segmentation - the prerequisite of numerous clinical needs -
has been significantly prospered by recent advances in convolutional neural
networks (CNNs). However, it exhibits general limitations on modeling explicit
long-range relation, and existing cures, resorting to building deep encoders
along with aggressive downsampling operations, leads to redundant deepened
networks and loss of localized details. Hence, the segmentation task awaits a
better solution to improve the efficiency of modeling global contexts while
maintaining a strong grasp of low-level details. In this paper, we propose a
novel parallel-in-branch architecture, TransFuse, to address this challenge.
TransFuse combines Transformers and CNNs in a parallel style, where both global
dependency and low-level spatial details can be efficiently captured in a much
shallower manner. Besides, a novel fusion technique - BiFusion module is
created to efficiently fuse the multi-level features from both branches.
Extensive experiments demonstrate that TransFuse achieves the newest
state-of-the-art results on both 2D and 3D medical image sets including polyp,
skin lesion, hip, and prostate segmentation, with significant parameter
decrease and inference speed improvement.

本文介绍了用于医学图像分割的 TransFuse 并行网络结构，将 Transformers 和 CNNs 并置，以有效地捕捉全局依赖关系和低级别空间细节，且使用了一种新的特征融合方法 —BiFusion 模块来融合两个分支的多级特征。实验结果表明，TransFuse 在医学图像分割任务中的性能优于目前最先进的技术，同时在参数数量和推断速度上也有显著改进。

TransFuse：将 Transformer 和 CNN 融合用于医学图像分割

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

Previous works have shown that contextual information can improve the
performance of neural machine translation (NMT). However, most existing
document-level NMT methods only consider a few number of previous sentences.
How to make use of the whole document as global contexts is still a challenge.
To address this issue, we hypothesize that a document can be represented as a
graph that connects relevant contexts regardless of their distances. We employ
several types of relations, including adjacency, syntactic dependency, lexical
consistency, and coreference, to construct the document graph. Then, we
incorporate both source and target graphs into the conventional Transformer
architecture with graph convolutional networks. Experiments on various NMT
benchmarks, including IWSLT English--French, Chinese-English, WMT
English--German and Opensubtitle English--Russian, demonstrate that using
document graphs can significantly improve the translation quality. Extensive
analysis verifies that the document graph is beneficial for capturing discourse
phenomena.

利用图形化表示法将文档作为全局语境，并将其整合到传统的变压器架构中，结合多种关系，包括邻接，句法依赖，词汇一致性和指代，能够显著提高神经机器翻译的性能。