We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io

我们引入了一个用于评估长距离三维点追踪任务的新基准，TAPVid-3D。通过利用现有素材，我们构建了一个包含4000多个真实世界视频的新基准，其涵盖了三个不同的数据源，涉及各种对象类型、运动模式以及室内外环境。我们提出了一系列度量方法来衡量TAP-3D任务的性能，并使用现有的追踪模型构建了竞争性基线。我们预计这个基准将指导我们更好地理解从单眼视频中精确获得三维运动和表面变形的能力。

TAPVid-3D：一个用于追踪三维空间中任意点的基准测试