We propose a novel test-time optimization approach for efficiently and
robustly tracking any pixel at any time in a video. The latest state-of-the-art
optimization-based tracking technique, OmniMotion, requires a prohibitively
long optimization time, rendering it impractical for downstream applications.
OmniMotion is sensitive to the choice of random seeds, leading to unstable
convergence. To improve efficiency and robustness, we introduce a novel
invertible deformation network, CaDeX++, which factorizes the function
representation into a local spatial-temporal feature grid and enhances the
expressivity of the coupling blocks with non-linear functions. While CaDeX++
incorporates a stronger geometric bias within its architectural design, it also
takes advantage of the inductive bias provided by the vision foundation models.
Our system utilizes monocular depth estimation to represent scene geometry and
enhances the objective by incorporating DINOv2 long-term semantics to regulate
the optimization process. Our experiments demonstrate a substantial improvement
in training speed (more than \textbf{10 times} faster), robustness, and
accuracy in tracking over the SoTA optimization-based method OmniMotion.

我们提出了一种新颖的测试时优化方法，可高效且鲁棒地跟踪视频中的任何像素。通过引入一种新颖的可逆形变网络 CaDeX ++，我们改进了效率和鲁棒性，并在架构设计中增加了更强的几何偏差，并利用视觉基础模型提供的归纳偏差。我们的系统利用单目深度估计来表示场景几何，并通过整合 DINOv2 的长期语义来调节优化过程，实验结果表明，与最先进的基于优化的方法 OmniMotion 相比，在训练速度（超过 10 倍）、鲁棒性和跟踪准确度方面得到了显著的改进。