Deep video models, for example, 3D CNNs or video transformers, have achieved
promising performance on sparse video tasks, i.e., predicting one result per
video. However, challenges arise when adapting existing deep video models to
dense video tasks, i.e., predicting one result per frame. Specifically, these
models are expensive for deployment, less effective when handling redundant
frames, and difficult to capture long-range temporal correlations. To overcome
these issues, we propose a Temporal Dilated Video Transformer (TDViT) that
consists of carefully designed temporal dilated transformer blocks (TDTB). TDTB
can efficiently extract spatiotemporal representations and effectively
alleviate the negative effect of temporal redundancy. Furthermore, by using
hierarchical TDTBs, our approach obtains an exponentially expanded temporal
receptive field and therefore can model long-range dynamics. Extensive
experiments are conducted on two different dense video benchmarks, i.e.,
ImageNet VID for video object detection and YouTube VIS for video instance
segmentation. Excellent experimental results demonstrate the superior
efficiency, effectiveness, and compatibility of our method. The code is
available at this https URL

我们提出了一种时态扩张视频变换器 (Temporal Dilated Video Transformer, TDViT)，通过使用层次化的时态扩张变换器块 (Temporal Dilated Transformer Blocks, TDTB) 来提取时空表示，并有效缓解时态冗余的负面影响，从而模拟长程动态。通过在两个不同的密集视频基准上进行广泛实验，即用于视频物体检测的 ImageNet VID 和用于视频实例分割的 YouTube VIS，出色的实验结果证明了我们方法的出色效率、有效性和兼容性。