Multiscale video transformers have been explored in a wide variety of vision
tasks. To date, however, the multiscale processing has been confined to the
encoder or decoder alone. We present a unified multiscale encoder-decoder
transformer that is focused on dense prediction tasks in videos. Multiscale
representation at both encoder and decoder yields key benefits of implicit
extraction of spatiotemporal features (i.e. without reliance on input optical
flow) as well as temporal consistency at encoding and coarseto-fine detection
for high-level (e.g. object) semantics to guide precise localization at
decoding. Moreover, we propose a transductive learning scheme through
many-to-many label propagation to provide temporally consistent predictions. We
showcase our Multiscale Encoder-Decoder Video Transformer (MED-VT) on Automatic
Video Object Segmentation (AVOS) and actor/action segmentation, where we
outperform state-of-the-art approaches on multiple benchmarks using only raw
images, without using optical flow.

本文提出了一种统一的多尺度编码 - 解码变形器，重点针对视频中的密集预测任务，利用编码器和解码器的多尺度表示提取时空特征以及进行精确的定位，同时采用多对多标签传播的转导学习方案，实现高精度视频对象分割和演员 / 动作分割。