It is now possible to estimate 3D human pose from monocular images with
off-the-shelf 3D pose estimators. However, many practical applications require
fine-grained absolute pose information for which multi-view cues and camera
calibration are necessary. Such multi-view recordings are laborious because
they require manual calibration, and are expensive when using dedicated
hardware. Our goal is full automation, which includes temporal synchronization,
as well as intrinsic and extrinsic camera calibration. This is done by using
persons in the scene as the calibration objects. Existing methods either
address only synchronization or calibration, assume one of the former as input,
or have significant limitations. A common limitation is that they only consider
single persons, which eases correspondence finding. We attain this generality
by partitioning the high-dimensional time and calibration space into a cascade
of subspaces and introduce tailored algorithms to optimize each efficiently and
robustly. The outcome is an easy-to-use, flexible, and robust motion capture
toolbox that we release to enable scientific applications, which we demonstrate
on diverse multi-view benchmarks. Project website:
this https URL

使用离线的 3D 姿势估计器，结合多视图线索和相机校准，自动化地获取 3D 人体姿态信息，包括时间同步和相机内外部校准，并通过将高维时间和校准空间划分为级联子空间以及引入定制算法来优化每个子空间，最终实现了易于使用、灵活且稳健的动作捕捉工具箱。

CasCalib：基于级联校准的来自稀疏非同步相机的动作捕捉

CasCalib: Cascaded Calibration for Motion Capture from Sparse  Unsynchronized Cameras

We tackle the task of reconstructing hand-object interactions from short
video clips. Given an input video, our approach casts 3D inference as a
per-video optimization and recovers a neural 3D representation of the object
shape, as well as the time-varying motion and hand articulation. While the
input video naturally provides some multi-view cues to guide 3D inference,
these are insufficient on their own due to occlusions and limited viewpoint
variations. To obtain accurate 3D, we augment the multi-view signals with
generic data-driven priors to guide reconstruction. Specifically, we learn a
diffusion network to model the conditional distribution of (geometric)
renderings of objects conditioned on hand configuration and category label, and
leverage it as a prior to guide the novel-view renderings of the reconstructed
scene. We empirically evaluate our approach on egocentric videos across 6
object categories, and observe significant improvements over prior single-view
and multi-view methods. Finally, we demonstrate our system's ability to
reconstruct arbitrary clips from YouTube, showing both 1st and 3rd person
interactions.

我们通过对短视频剪辑进行手物体交互的重建任务。以输入视频为基础，我们的方法将 3D 推断视为每个视频的优化，并恢复物体形状的神经 3D 表示，以及时间变化的动作和手关节。尽管输入视频自然地提供了一些多视角线索来指导 3D 推断，但由于遮挡和有限的视角变化，这些线索本身并不足够。为了获得准确的 3D 结果，我们使用通用的数据驱动先验来引导重建过程中的多视角信号。具体来说，我们学习了一个扩散网络，来建模基于手配置和类别标签的对象（几何）渲染的条件分布，并利用它作为先验来指导重建场景的新视角渲染。我们在 6 个物体类别的自我中心视频上对我们的方法进行了实证评估，并观察到相较于先前的单视角和多视角方法有显著的改进。最后，我们展示了我们的系统能够从 YouTube 中重建任意剪辑，展示了第一人称和第三人称的交互。

基于扩散导向的日常手 - 物交互重建剪辑

Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction  Clips

Multi-frame depth estimation generally achieves high accuracy relying on the
multi-view geometric consistency. When applied in dynamic scenes, e.g.,
autonomous driving, this consistency is usually violated in the dynamic areas,
leading to corrupted estimations. Many multi-frame methods handle dynamic areas
by identifying them with explicit masks and compensating the multi-view cues
with monocular cues represented as local monocular depth or features. The
improvements are limited due to the uncontrolled quality of the masks and the
underutilized benefits of the fusion of the two types of cues. In this paper,
we propose a novel method to learn to fuse the multi-view and monocular cues
encoded as volumes without needing the heuristically crafted masks. As unveiled
in our analyses, the multi-view cues capture more accurate geometric
information in static areas, and the monocular cues capture more useful
contexts in dynamic areas. To let the geometric perception learned from
multi-view cues in static areas propagate to the monocular representation in
dynamic areas and let monocular cues enhance the representation of multi-view
cost volume, we propose a cross-cue fusion (CCF) module, which includes the
cross-cue attention (CCA) to encode the spatially non-local relative
intra-relations from each source to enhance the representation of the other.
Experiments on real-world datasets prove the significant effectiveness and
generalization ability of the proposed method.

本文提出了一种新的方法，能够在显式掩模和单眼线索的帮助下，利用体积编码融合多视图几何感知与单眼图像中的上下文信息，从而实现对多帧深度估计的更精确的处理。实验结果证明了这种方法在真实世界数据集中的有效性与通用性。