Recently, transformer-based methods have dominated 3D instance segmentation,
where mask attention is commonly involved. Specifically, object queries are
guided by the initial instance masks in the first cross-attention, and then
iteratively refine themselves in a similar manner. However, we observe that the
mask-attention pipeline usually leads to slow convergence due to low-recall
initial instance masks. Therefore, we abandon the mask attention design and
resort to an auxiliary center regression task instead. Through center
regression, we effectively overcome the low-recall issue and perform
cross-attention by imposing positional prior. To reach this goal, we develop a
series of position-aware designs. First, we learn a spatial distribution of 3D
locations as the initial position queries. They spread over the 3D space
densely, and thus can easily capture the objects in a scene with a high recall.
Moreover, we present relative position encoding for the cross-attention and
iterative refinement for more accurate position queries. Experiments show that
our approach converges 4x faster than existing work, sets a new state of the
art on ScanNetv2 3D instance segmentation benchmark, and also demonstrates
superior performance across various datasets. Code and models are available at
this https URL

通过引入位置感知设计，利用辅助中心回归任务替代掩码注意力设计，实现了在 3D 实例分割中更快的收敛速度和更准确的位置查询，并在多种数据集上展现出优越的性能。

无掩模关注的 Transformer 用于 3D 实例分割

Mask-Attention-Free Transformer for 3D Instance Segmentation

Today the pre-trained language models achieve great success for question
generation (QG) task and significantly outperform traditional
sequence-to-sequence approaches. However, the pre-trained models treat the
input passage as a flat sequence and are thus not aware of the text structure
of input passage. For QG task, we model text structure as answer position and
syntactic dependency, and propose answer localness modeling and syntactic mask
attention to address these limitations. Specially, we present localness
modeling with a Gaussian bias to enable the model to focus on answer-surrounded
context, and propose a mask attention mechanism to make the syntactic structure
of input passage accessible in question generation process. Experiments on
SQuAD dataset show that our proposed two modules improve performance over the
strong pre-trained model ProphetNet, and combing them together achieves very
competitive results with the state-of-the-art pre-trained model.

研究表明，采用预训练模型生成问题 (QG) 任务可以显著优于传统的序列对序列方法，但是这些模型对于输入的文章缺乏文本结构的认识。为了克服这个问题，我们将文本结构模拟为回答位置和句法依赖性，提出回答局部性建模和句法遮盖自注意力机制。实验表明，这两个模块可以改进预训练模型 ProphetNet 的性能，并将它们结合在一起可以达到现有最先进预训练模型的竞争性结果。