We propose NeRFiller, an approach that completes missing portions of a 3D
capture via generative 3D inpainting using off-the-shelf 2D visual generative
models. Often parts of a captured 3D scene or object are missing due to mesh
reconstruction failures or a lack of observations (e.g., contact regions, such
as the bottom of objects, or hard-to-reach areas). We approach this challenging
3D inpainting problem by leveraging a 2D inpainting diffusion model. We
identify a surprising behavior of these models, where they generate more 3D
consistent inpaints when images form a 2$\times$2 grid, and show how to
generalize this behavior to more than four images. We then present an iterative
framework to distill these inpainted regions into a single consistent 3D scene.
In contrast to related works, we focus on completing scenes rather than
deleting foreground objects, and our approach does not require tight 2D object
masks or text. We compare our approach to relevant baselines adapted to our
setting on a variety of scenes, where NeRFiller creates the most 3D consistent
and plausible scene completions. Our project page is at
this https URL

提出了一种名为 NeRFiller 的方法，使用现成的 2D 视觉生成模型进行基于生成式 3D 修复的 3D 捕捉的缺失部分的填充。与相关工作相比，我们的方法侧重于完成场景，而不是删除前景对象，并且不需要紧贴 2D 对象掩码或文本。

NeRFiller: 通过生成式 3D 修补完成场景

NeRFiller: Completing Scenes via Generative 3D Inpainting

Since humans interact with diverse objects every day, the holistic 3D capture
of these interactions is important to understand and model human behaviour.
However, most existing methods for hand-object reconstruction from RGB either
assume pre-scanned object templates or heavily rely on limited 3D hand-object
data, restricting their ability to scale and generalize to more unconstrained
interaction settings. To this end, we introduce HOLD -- the first
category-agnostic method that reconstructs an articulated hand and object
jointly from a monocular interaction video. We develop a compositional
articulated implicit model that can reconstruct disentangled 3D hand and object
from 2D images. We also further incorporate hand-object constraints to improve
hand-object poses and consequently the reconstruction quality. Our method does
not rely on 3D hand-object annotations while outperforming fully-supervised
baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we
qualitatively show its robustness in reconstructing from in-the-wild videos.
Code: this https URL

通过使用单目交互视频，我们提出了 HOLD，它是第一个无需 3D 手物体注释就能从中复原出联合的手和物体的方法，通过使用一个组合的隐式模型，能够从 2D 图像中复原出解缠细分的 3D 手和物体，并进一步结合手物体约束来改善复原质量，在实验室和入迷环境中优于全监督基线方法。

HOLD：从视频中无类别限制地重建互动的手部和物体的 3D 模型

HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and  Objects from Video

With the advent of affordable depth sensors, 3D capture becomes more and more
ubiquitous and already has made its way into commercial products. Yet,
capturing the geometry or complete shapes of everyday objects using scanning
devices (e.g. Kinect) still comes with several challenges that result in noise
or even incomplete shapes. Recent success in deep learning has shown how to
learn complex shape distributions in a data-driven way from large scale 3D CAD
Model collections and to utilize them for 3D processing on volumetric
representations and thereby circumventing problems of topology and
tessellation. Prior work has shown encouraging results on problems ranging from
shape completion to recognition. We provide an analysis of such approaches and
discover that training as well as the resulting representation are strongly and
unnecessarily tied to the notion of object labels. Thus, we propose a full
convolutional volumetric auto encoder that learns volumetric representation
from noisy data by estimating the voxel occupancy grids. The proposed method
outperforms prior work on challenging tasks like denoising and shape
completion. We also show that the obtained deep embedding gives competitive
performance when used for classification and promising results for shape
interpolation.

通过提出一种新的全卷积体积自动编码器，有效实现了从嘈杂的 3D 数据中提取体积表示，并在噪声去除和形状完成等多个任务上超越了以往的技术，同时在分类和形状插值方面也得到了不错的结果。