We tackle the long-standing challenge of reconstructing 3D structures and
camera positions from videos. The problem is particularly hard when objects are
transformed in a non-rigid way. Current approaches to this problem make
unrealistic assumptions or require a long optimization time.
We present TracksTo4D, a novel deep learning-based approach that enables
inferring 3D structure and camera positions from dynamic content originating
from in-the-wild videos using a single feed-forward pass on a sparse point
track matrix. To achieve this, we leverage recent advances in 2D point tracking
and design an equivariant neural architecture tailored for directly processing
2D point tracks by leveraging their symmetries. TracksTo4D is trained on a
dataset of in-the-wild videos utilizing only the 2D point tracks extracted from
the videos, without any 3D supervision. Our experiments demonstrate that
TracksTo4D generalizes well to unseen videos of unseen semantic categories at
inference time, producing equivalent results to state-of-the-art methods while
significantly reducing the runtime compared to other baselines.

通过一次稀疏点轨迹矩阵的前向传递，我们提出了 TracksTo4D—— 一种新颖的基于深度学习的方法，用于从野外视频中推断出来自动态内容的三维结构和相机位置。

从普通视频学习非刚性 SfM 的先验知识

Learning Priors for Non Rigid SfM from Casual Videos

Multi-image super-resolution (MISR) allows to increase the spatial resolution
of a low-resolution (LR) acquisition by combining multiple images carrying
complementary information in the form of sub-pixel offsets in the scene
sampling, and can be significantly more effective than its single-image
counterpart. Its main difficulty lies in accurately registering and fusing the
multi-image information. Currently studied settings, such as burst photography,
typically involve assumptions of small geometric disparity between the LR
images and rely on optical flow for image registration. We study a MISR method
that can increase the resolution of sets of images acquired with arbitrary, and
potentially wildly different, camera positions and orientations, generalizing
the currently studied MISR settings. Our proposed model, called EpiMISR, moves
away from optical flow and explicitly uses the epipolar geometry of the
acquisition process, together with transformer-based processing of radiance
feature fields to substantially improve over state-of-the-art MISR methods in
presence of large disparities in the LR images.

多图像超分辨率 (MISR) 方法以子像素偏移的形式结合多张图像中的互补信息来提高低分辨率 (LR) 图像的空间分辨率，在准确注册和融合多图像信息方面存在困难。本研究提出的 EpiMISR 模型通过使用采集过程的极线几何以及基于 Transformer 的辐射特征场进行处理，在存在大的 LR 图像视差的情况下明显改进了目前最先进的 MISR 方法。