We propose an approach for reconstructing free-moving object from a monocular
RGB video. Most existing methods either assume scene prior, hand pose prior,
object category pose prior, or rely on local optimization with multiple
sequence segments. We propose a method that allows free interaction with the
object in front of a moving camera without relying on any prior, and optimizes
the sequence globally without any segments. We progressively optimize the
object shape and pose simultaneously based on an implicit neural
representation. A key aspect of our method is a virtual camera system that
reduces the search space of the optimization significantly. We evaluate our
method on the standard HO3D dataset and a collection of egocentric RGB
sequences captured with a head-mounted device. We demonstrate that our approach
outperforms most methods significantly, and is on par with recent techniques
that assume prior information.

我们提出了一种从单目 RGB 视频中重建自由移动物体的方法，无需任何先验信息，通过基于隐式神经表示同时优化对象的形状和姿态，其中方法的关键是一个虚拟摄像机系统，显著减小了优化的搜索空间。通过在 HO3D 数据集和佩戴在头部设备上捕获的主观 RGB 序列上的评估，我们证明了该方法明显优于大多数方法，并且与假设先验信息的最新技术相媲美。

基于虚拟相机的自由移动物体重建与姿态估计

Free-Moving Object Reconstruction and Pose Estimation with Virtual  Camera

Dynamic Neural Radiance Field (NeRF) is a powerful algorithm capable of
rendering photo-realistic novel view images from a monocular RGB video of a
dynamic scene. Although it warps moving points across frames from the
observation spaces to a common canonical space for rendering, dynamic NeRF does
not model the change of the reflected color during the warping. As a result,
this approach often fails drastically on challenging specular objects in
motion. We address this limitation by reformulating the neural radiance field
function to be conditioned on surface position and orientation in the
observation space. This allows the specular surface at different poses to keep
the different reflected colors when mapped to the common canonical space.
Additionally, we add the mask of moving objects to guide the deformation field.
As the specular surface changes color during motion, the mask mitigates the
problem of failure to find temporal correspondences with only RGB supervision.
We evaluate our model based on the novel view synthesis quality with a
self-collected dataset of different moving specular objects in realistic
environments. The experimental results demonstrate that our method
significantly improves the reconstruction quality of moving specular objects
from monocular RGB videos compared to the existing NeRF models. Our code and
data are available at the project website this https URL

本文提出了一种基于动态神经辐射场的算法，通过表面的位置和朝向在观测空间中进行条件化，像素的渲染基于不同姿态下的表面反射的颜色来实现遮挡的效果，有效地改善了单目 RGB 视频中移动高光物体的重建质量。

动态高光物体的神经辐射场（NeRF-DS）

NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects

We present a method to capture temporally coherent dynamic clothing
deformation from a monocular RGB video input. In contrast to the existing
literature, our method does not require a pre-scanned personalized mesh
template, and thus can be applied to in-the-wild videos. To constrain the
output to a valid deformation space, we build statistical deformation models
for three types of clothing: T-shirt, short pants and long pants. A
differentiable renderer is utilized to align our captured shapes to the input
frames by minimizing the difference in both silhouette, segmentation, and
texture. We develop a UV texture growing method which expands the visible
texture region of the clothing sequentially in order to minimize drift in
deformation tracking. We also extract fine-grained wrinkle detail from the
input videos by fitting the clothed surface to the normal maps estimated by a
convolutional neural network. Our method produces temporally coherent
reconstruction of body and clothing from monocular video. We demonstrate
successful clothing capture results from a variety of challenging videos.
Extensive quantitative experiments demonstrate the effectiveness of our method
on metrics including body pose error and surface reconstruction error of the
clothing.

本文提出了一种从单目 RGB 视频输入中捕捉时间连贯的动态服装变形的方法，该方法不需要预先扫描个性化网格模板，可以用于野外视频，并建立三种类型的服装的统计变形模型，利用可微分渲染器将捕获到的形状与输入帧对齐，并提取出细节纹理等信息。经过大量实验表明，该方法可以成功地从各种具有挑战性的视频中捕获服装，同时也能够在衣服表面重建误差等指标上达到有效的效果。