We present Viewset Diffusion: a framework for training image-conditioned 3D
generative models from 2D data. Image-conditioned 3D generative models allow us
to address the inherent ambiguity in single-view 3D reconstruction. Given one
image of an object, there is often more than one possible 3D volume that
matches the input image, because a single image never captures all sides of an
object. Deterministic models are inherently limited to producing one possible
reconstruction and therefore make mistakes in ambiguous settings. Modelling
distributions of 3D shapes is challenging because 3D ground truth data is often
not available. We propose to solve the issue of data availability by training a
diffusion model which jointly denoises a multi-view image set.We constrain the
output of Viewset Diffusion models to a single 3D volume per image set,
guaranteeing consistent geometry. Training is done through reconstruction
losses on renderings, allowing training with only three images per object. Our
design of architecture and training scheme allows our model to perform 3D
generation and generative, ambiguity-aware single-view reconstruction in a
feed-forward manner. Project page: szymanowiczs.github.io/viewset-diffusion.

借助 Viewset Diffusion 框架，可以从 2D 数据中训练图像条件化的 3D 生成模型，从而解决单视图 3D 重建中的歧义问题，并通过对多视图图像集的去噪扩展了 3D 真实数据的可用性，通过仅渲染 3 张图片，我们的模型可以执行 3D 生成和单视图重建。

视图集扩散：从二维数据生成 (0-) 图像条件下的三维生成模型

Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D  Data

Layout generation is a novel task in computer vision, which combines the
challenges in both object localization and aesthetic appraisal, widely used in
advertisements, posters, and slides design. An accurate and pleasant layout
should consider both the intra-domain relationship within layout elements and
the inter-domain relationship between layout elements and the image. However,
most previous methods simply focus on image-content-agnostic layout generation,
without leveraging the complex visual information from the image. To this end,
we explore a novel paradigm entitled image-conditioned layout generation, which
aims to add text overlays to an image in a semantically coherent manner.
Specifically, we propose an Image-Conditioned Variational Transformer (ICVT)
that autoregressively generates various layouts in an image. First,
self-attention mechanism is adopted to model the contextual relationship within
layout elements, while cross-attention mechanism is used to fuse the visual
information of conditional images. Subsequently, we take them as building
blocks of conditional variational autoencoder (CVAE), which demonstrates
appealing diversity. Second, in order to alleviate the gap between layout
elements domain and visual domain, we design a Geometry Alignment module, in
which the geometric information of the image is aligned with the layout
representation. In addition, we construct a large-scale advertisement poster
layout designing dataset with delicate layout and saliency map annotations.
Experimental results show that our model can adaptively generate layouts in the
non-intrusive area of the image, resulting in a harmonious layout design.

本文探讨了一种新的图像条件布局生成的范式，通过采用自我关注机制和交叉关注机制来建立布局元素间的关系并结合条件图像的视觉信息生成各种布局，其中几何对齐模块可以消除元素域和视觉域之间的差距，在大规模广告海报布局设计数据集上进行的实验结果表明，我们的模型可以自适应地生成非侵入式区域的布局，从而实现和谐的布局设计。