Recovering dense and long-range pixel motion in videos is a challenging
problem. Part of the difficulty arises from the 3D-to-2D projection process,
leading to occlusions and discontinuities in the 2D motion domain. While 2D
motion can be intricate, we posit that the underlying 3D motion can often be
simple and low-dimensional. In this work, we propose to estimate point
trajectories in 3D space to mitigate the issues caused by image projection. Our
method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth
estimators, represents the 3D content of each frame efficiently using a
triplane representation, and performs iterative updates using a transformer to
estimate 3D trajectories. Tracking in 3D allows us to leverage
as-rigid-as-possible (ARAP) constraints while simultaneously learning a
rigidity embedding that clusters pixels into different rigid parts. Extensive
evaluation shows that our approach achieves state-of-the-art tracking
performance both qualitatively and quantitatively, particularly in challenging
scenarios such as out-of-plane rotation.

通过估计 3D 空间中的点轨迹以减轻由图像投影引起的问题，本研究提出了一种名为 SpatialTracker 的方法，使用单目深度估计器将 2D 像素提升为 3D，并使用三平面表示高效地呈现每帧的 3D 内容，并使用变换器进行迭代更新以估计 3D 轨迹，通过在 3D 空间中进行跟踪，充分利用几乎刚性的约束，同时学习将像素分成不同刚性部件的刚性嵌入，广泛的评估结果显示，我们的方法在定性和定量上都实现了最先进的跟踪性能，特别是在包括平面外旋转等具有挑战性的场景中。