Diffusion models have emerged as the best approach for generative modeling of
2D images. Part of their success is due to the possibility of training them on
millions if not billions of images with a stable learning objective. However,
extending these models to 3D remains difficult for two reasons. First, finding
a large quantity of 3D training data is much more complex than for 2D images.
Second, while it is conceptually trivial to extend the models to operate on 3D
rather than 2D grids, the associated cubic growth in memory and compute
complexity makes this infeasible. We address the first challenge by introducing
a new diffusion setup that can be trained, end-to-end, with only posed 2D
images for supervision; and the second challenge by proposing an image
formation model that decouples model memory from spatial memory. We evaluate
our method on real-world data, using the CO3D dataset which has not been used
to train 3D generative models before. We show that our diffusion models are
scalable, train robustly, and are competitive in terms of sample quality and
fidelity to existing approaches for 3D generative modeling.

本研究提出了一种新的扩散设置，可以仅通过 2D 图像监督就可以进行端到端的训练，同时提出的图像形成模型将模型内存与空间内存分离，以应对 3D 扩散模型在训练上遇到的问题，并在 CO3D 数据集上进行的实验表明，该方法是可扩展的，具有稳健性，与现有的 3D 生成建模方法在样本质量和保真度方面相当竞争。

HOLODIFFUSION：使用 2D 图像训练 3D 扩散模型

HOLODIFFUSION: Training a 3D Diffusion Model using 2D Images

We present a simple and effective image super-resolution algorithm that
imposes an image formation constraint on the deep neural networks via pixel
substitution. The proposed algorithm first uses a deep neural network to
estimate intermediate high-resolution images, blurs the intermediate images
using known blur kernels, and then substitutes values of the pixels at the
un-decimated positions with those of the corresponding pixels from the
low-resolution images. The output of the pixel substitution process strictly
satisfies the image formation model and is further refined by the same deep
neural network in a cascaded manner. The proposed framework is trained in an
end-to-end fashion and can work with existing feed-forward deep neural networks
for super-resolution and converges fast in practice. Extensive experimental
results show that the proposed algorithm performs favorably against
state-of-the-art methods.

本篇论文提出了一种简单有效的基于像素替换的图像超分辨率算法，该算法使用深度神经网络估计中间高分辨率图像，并通过从低分辨率图像替换像素值来满足图像形成模型，并进行级联细化。实验结果显示，所提出的算法在图像超分辨率方面的性能优于现有的同类算法。

基于图像形成模型的深度图像超分辨率

Image Formation Model Guided Deep Image Super-Resolution

Generative reconstruction methods compute the 3D configuration (such as pose
and/or geometry) of a shape by optimizing the overlap of the projected 3D shape
model with images. Proper handling of occlusions is a big challenge, since the
visibility function that indicates if a surface point is seen from a camera can
often not be formulated in closed form, and is in general discrete and
non-differentiable at occlusion boundaries. We present a new scene
representation that enables an analytically differentiable closed-form
formulation of surface visibility. In contrast to previous methods, this yields
smooth, analytically differentiable, and efficient to optimize pose similarity
energies with rigorous occlusion handling, fewer local minima, and
experimentally verified improved convergence of numerical optimization. The
underlying idea is a new image formation model that represents opaque objects
by a translucent medium with a smooth Gaussian density distribution which turns
visibility into a smooth phenomenon. We demonstrate the advantages of our
versatile scene model in several generative pose estimation problems, namely
marker-less multi-object pose estimation, marker-less human motion capture with
few cameras, and image-based 3D geometry estimation.

本论文提出了一种新的场景表示方法，它能够处理形态重建中存在的挑战，例如遮挡，以及在多种生成姿态估计问题中的优越表现。