We tackle the long-standing challenge of reconstructing 3D structures and
camera positions from videos. The problem is particularly hard when objects are
transformed in a non-rigid way. Current approaches to this problem make
unrealistic assumptions or require a long optimization time.
We present TracksTo4D, a novel deep learning-based approach that enables
inferring 3D structure and camera positions from dynamic content originating
from in-the-wild videos using a single feed-forward pass on a sparse point
track matrix. To achieve this, we leverage recent advances in 2D point tracking
and design an equivariant neural architecture tailored for directly processing
2D point tracks by leveraging their symmetries. TracksTo4D is trained on a
dataset of in-the-wild videos utilizing only the 2D point tracks extracted from
the videos, without any 3D supervision. Our experiments demonstrate that
TracksTo4D generalizes well to unseen videos of unseen semantic categories at
inference time, producing equivalent results to state-of-the-art methods while
significantly reducing the runtime compared to other baselines.

通过一次稀疏点轨迹矩阵的前向传递，我们提出了 TracksTo4D—— 一种新颖的基于深度学习的方法，用于从野外视频中推断出来自动态内容的三维结构和相机位置。

从普通视频学习非刚性 SfM 的先验知识

Learning Priors for Non Rigid SfM from Casual Videos

This paper presents DriveTrack, a new benchmark and data generation framework
for long-range keypoint tracking in real-world videos. DriveTrack is motivated
by the observation that the accuracy of state-of-the-art trackers depends
strongly on visual attributes around the selected keypoints, such as texture
and lighting. The problem is that these artifacts are especially pronounced in
real-world videos, but these trackers are unable to train on such scenes due to
a dearth of annotations. DriveTrack bridges this gap by building a framework to
automatically annotate point tracks on autonomous driving datasets. We release
a dataset consisting of 1 billion point tracks across 24 hours of video, which
is seven orders of magnitude greater than prior real-world benchmarks and on
par with the scale of synthetic benchmarks. DriveTrack unlocks new use cases
for point tracking in real-world videos. First, we show that fine-tuning
keypoint trackers on DriveTrack improves accuracy on real-world scenes by up to
7%. Second, we analyze the sensitivity of trackers to visual artifacts in real
scenes and motivate the idea of running assistive keypoint selectors alongside
trackers.

DriveTrack 是一个新的基准和数据生成框架，用于实现长距离实景关键点跟踪。该框架自动注释自动驾驶数据集上的关键点轨迹，释放了关键点在实景视频中的新用途，并提高了关键点跟踪器在实世界场景中的准确性。