Methods for video motion prediction either estimate jointly the instantaneous
motion of all points in a given video frame using optical flow or independently
track the motion of individual points throughout the video. The latter is true
even for powerful deep-learning methods that can track points through
occlusions. Tracking points individually ignores the strong correlation that
can exist between the points, for instance, because they belong to the same
physical object, potentially harming performance. In this paper, we thus
propose CoTracker, an architecture that jointly tracks multiple points
throughout an entire video. This architecture combines several ideas from the
optical flow and tracking literature in a new, flexible and powerful design. It
is based on a transformer network that models the correlation of different
points in time via specialised attention layers. The transformer iteratively
updates an estimate of several trajectories. It can be applied in a
sliding-window manner to very long videos, for which we engineer an unrolled
training loop. It can track from one to several points jointly and supports
adding new points to track at any time. The result is a flexible and powerful
tracking algorithm that outperforms state-of-the-art methods in almost all
benchmarks.

该论文提出了一种名为 CoTracker 的架构，它利用了光流和跟踪文献中的几个思想，基于变压器网络来模拟不同点在时间上的相关性，可以在一个视频中联合跟踪多个点，支持在任何时候添加新的跟踪点。该算法在几乎所有基准测试中都优于最先进的方法。