Diffusion-based image generation models such as DALL-E 3 and Stable
Diffusion-XL demonstrate remarkable capabilities in generating images with
realistic and unique compositions. Yet, these models are not robust in
precisely reasoning about physical and spatial configurations of objects,
especially when instructed with unconventional, thereby out-of-distribution
descriptions, such as "a chair with five legs". In this paper, we propose a
language agent with chain-of-3D-thoughts (L3GO), an inference-time approach
that can reason about part-based 3D mesh generation of unconventional objects
that current data-driven diffusion models struggle with. More concretely, we
use large language models as agents to compose a desired object via
trial-and-error within the 3D simulation environment. To facilitate our
investigation, we develop a new benchmark, Unconventionally Feasible Objects
(UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender
where language agents can build and compose atomic building blocks via API
calls. Human and automatic GPT-4V evaluations show that our approach surpasses
the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D
mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our
approach outperforms other state-of-the-art text-to-2D image and text-to-3D
models based on human evaluation.

通过 L3GO 模型，利用大型语言模型作为代理，使用 3D 仿真环境进行试错式组合，推理和生成 Diffusion-based 图像模型中难以处理的非传统对象的基于部件的 3D 网格生成。在 ShapeNet 和 UFO 基准上，我们的方法在人工和自动评估中表现优于标准 GPT-4 和其他语言代理。

L3GO: 基于三维思维链的语言智能代理生成非常规物体

L3GO: Language Agents with Chain-of-3D-Thoughts for Generating  Unconventional Objects

We introduce Cap3D, an automatic approach for generating descriptive text for
3D objects. This approach utilizes pretrained models from image captioning,
image-text alignment, and LLM to consolidate captions from multiple views of a
3D asset, completely side-stepping the time-consuming and costly process of
manual annotation. We apply Cap3D to the recently introduced large-scale 3D
dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted
using 41k human annotations from the same dataset, demonstrates that Cap3D
surpasses human-authored descriptions in terms of quality, cost, and speed.
Through effective prompt engineering, Cap3D rivals human performance in
generating geometric descriptions on 17k collected annotations from the ABO
dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions,
and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shape-E,
and DreamFusion.

Cap3D 是一种自动生成三维物体描述的方法，利用图像字幕、图像文本对齐和 LLM 的预训练模型，从多个视角的 3D 资源中整合标题，既提高了效率又降低了成本，并在大规模 3D 数据集 Objaverse 上实现了 660k 3D 文本对。通过有效的提示工程，Cap3D 在 ABO 数据集的 17k 个收集的注释中产生几何描述并与人类表现不相上下。在 Cap3D 和人类字幕上调整 Text-to-3D 模型，并展示 Cap3D 优于 Point-E、Shape-E 和 DreamFusion，是一项值得关注的自动化方法。