Recently, text-to-image generation has exhibited remarkable advancements,
with the ability to produce visually impressive results. In contrast,
text-to-3D generation has not yet reached a comparable level of quality.
Existing methods primarily rely on text-guided score distillation sampling
(SDS), and they encounter difficulties in transferring 2D attributes of the
generated images to 3D content. In this work, we aim to develop an effective 3D
generative model capable of synthesizing high-resolution textured meshes by
leveraging both textual and image information. To this end, we introduce
Guide3D, a zero-shot text-and-image-guided generative model for 3D avatar
generation based on diffusion models. Our model involves (1) generating
sparse-view images of a text-consistent character using diffusion models, and
(2) jointly optimizing multi-resolution differentiable marching tetrahedral
grids with pixel-aligned image features. We further propose a similarity-aware
feature fusion strategy for efficiently integrating features from different
views. Moreover, we introduce two novel training objectives as an alternative
to calculating SDS, significantly enhancing the optimization process. We
thoroughly evaluate the performance and components of our framework, which
outperforms the current state-of-the-art in producing topologically and
structurally correct geometry and high-resolution textures. Guide3D enables the
direct transfer of 2D-generated images to the 3D space. Our code will be made
publicly available.

通过结合文字和图像信息，利用扩散模型，我们开发了一种零样本的三维生成模型，能够合成高分辨率纹理网格，实现了将二维图像直接转化为三维空间的功能。