Diffusion models have emerged as a powerful method of generative modeling
across a range of fields, capable of producing stunning photo-realistic images
from natural language descriptions. However, these models lack explicit control
over the 3D structure of the objects in the generated images. In this paper, we
propose a novel method that incorporates 3D geometry control into diffusion
models, making them generate even more realistic and diverse images. To achieve
this, our method exploits ControlNet, which extends diffusion models by using
visual prompts in addition to text prompts. We generate images of 3D objects
taken from a 3D shape repository (e.g., ShapeNet and Objaverse), render them
from a variety of poses and viewing directions, compute the edge maps of the
rendered images, and use these edge maps as visual prompts to generate
realistic images. With explicit 3D geometry control, we can easily change the
3D structures of the objects in the generated images and obtain ground-truth 3D
annotations automatically. This allows us to use the generated images to
improve a lot of vision tasks, e.g., classification and 3D pose estimation, in
both in-distribution (ID) and out-of-distribution (OOD) settings. We
demonstrate the effectiveness of our method through extensive experiments on
ImageNet-50, ImageNet-R, PASCAL3D+, ObjectNet3D, and OOD-CV datasets. The
results show that our method significantly outperforms existing methods across
multiple benchmarks (e.g., 4.6 percentage points on ImageNet-50 using ViT and
3.5 percentage points on PASCAL3D+ and ObjectNet3D using NeMo).

本篇论文提出了一种利用 3D geometry control 和 visual prompts 的扩展 diffusion models 生成图像的方法，可应用于计算机视觉任务并经过多个数据集的多次实验验证其效果。