Recently, video transformers have shown great success in video understanding,
exceeding CNN performance; yet existing video transformer models do not
explicitly model objects, although objects can be essential for recognizing
actions. In this work, we present Object-Region Video Transformers (ORViT), an
\emph{object-centric} approach that extends video transformer layers with a
block that directly incorporates object representations. The key idea is to
fuse object-centric representations starting from early layers and propagate
them into the transformer-layers, thus affecting the spatio-temporal
representations throughout the network. Our ORViT block consists of two
object-level streams: appearance and dynamics. In the appearance stream, an
"Object-Region Attention" module applies self-attention over the patches and
\emph{object regions}. In this way, visual object regions interact with uniform
patch tokens and enrich them with contextualized object information. We further
model object dynamics via a separate "Object-Dynamics Module", which captures
trajectory interactions, and show how to integrate the two streams. We evaluate
our model on four tasks and five datasets: compositional and few-shot action
recognition on SomethingElse, spatio-temporal action detection on AVA, and
standard action recognition on Something-Something V2, Diving48 and
Epic-Kitchen100. We show strong performance improvement across all tasks and
datasets considered, demonstrating the value of a model that incorporates
object representations into a transformer architecture. For code and pretrained
models, visit the project page at https://roeiherz.github.io/ORViT/

本文提出了 Object-Region Video Transformers (ORViT)—— 一种基于对象的视觉转换方法，通过融合早期层的对象间信息直接影响整个网络的时空表示。我们的模型通过面向对象的自我关注和分离对象的动态特征来捕获物体和其在视频中的动态交互，取得了在多项任务和数据集上的显著性能提升。