Finetuning a pretrained backbone in the encoder part of an image transformer
network has been the traditional approach for the semantic segmentation task.
However, such an approach leaves out the semantic context that an image
provides during the encoding stage. This paper argues that incorporating
semantic information of the image into pretrained hierarchical
transformer-based backbones while finetuning improves the performance
considerably. To achieve this, we propose SeMask, a simple and effective
framework that incorporates semantic information into the encoder with the help
of a semantic attention operation. In addition, we use a lightweight semantic
decoder during training to provide supervision to the intermediate semantic
prior maps at every stage. Our experiments demonstrate that incorporating
semantic priors enhances the performance of the established hierarchical
encoders with a slight increase in the number of FLOPs. We provide empirical
proof by integrating SeMask into Swin Transformer and Mix Transformer backbones
as our encoder paired with different decoders. Our framework achieves a new
state-of-the-art of 58.25% mIoU on the ADE20K dataset and improvements of over
3% in the mIoU metric on the Cityscapes dataset. The code and checkpoints are
publicly available at
this https URL .

本文提出了 SeMask 框架，将语义信息嵌入预训练分层 Transformer 模块的编码器中以提高模型性能，并使用轻量级语义解码器进行训练。实验结果表明，嵌入语义先验可以显著提高模型效果，并且与 Swin Transformer 和 Mix Transformer 等模型相结合，达到了 58.25％的 mIoU 表现。

SeMask: 用于语义分割的语义掩蔽变形器

SeMask: Semantically Masked Transformers for Semantic Segmentation

Temporal action localization aims to localize starting and ending time with
action category. Limited by GPU memory, mainstream methods pre-extract features
for each video. Therefore, feature quality determines the upper bound of
detection performance. In this technical report, we explored classic
convolution-based backbones and the recent surge of transformer-based
backbones. We found that the transformer-based methods can achieve better
classification performance than convolution-based, but they cannot generate
accuracy action proposals. In addition, extracting features with larger frame
resolution to reduce the loss of spatial information can also effectively
improve the performance of temporal action localization. Finally, we achieve
42.42% in terms of mAP on validation set with a single SlowFast feature by a
simple combination: BMN+TCANet, which is 1.87% higher than the result of 2020's
multi-model ensemble. Finally, we achieve Rank 1st on the CVPR2021 HACS
supervised Temporal Action Localization Challenge.

本文研究了时域动作定位的方法，发现基于变压器的方法可以实现更好的分类性能，但不能生成准确的动作提案，最后通过以较高的帧分辨率提取特征来提高时域动作定位的性能，最终在 CVPR2021HACS 挑战中取得了第一名。