The current advances in generative AI for learning large neural network
models with the capability to produce essays, images, music and even 3D assets
from text prompts create opportunities for a manifold of disciplines. In the
present paper, we study the potential of deep text-to-3D models in the
engineering domain, with focus on the chances and challenges when integrating
and interacting with 3D assets in computational simulation-based design
optimization. In contrast to traditional design optimization of 3D geometries
that often searches for the optimum designs using numerical representations,
such as B-Spline surface or deformation parameters in vehicle aerodynamic
optimization, natural language challenges the optimization framework by
requiring a different interpretation of variation operators while at the same
time may ease and motivate the human user interaction. Here, we propose and
realize a fully automated evolutionary design optimization framework using
Shap-E, a recently published text-to-3D asset network by OpenAI, in the context
of aerodynamic vehicle optimization. For representing text prompts in the
evolutionary optimization, we evaluate (a) a bag-of-words approach based on
prompt templates and Wordnet samples, and (b) a tokenisation approach based on
prompt templates and the byte pair encoding method from GPT4. Our main findings
from the optimizations indicate that, first, it is important to ensure that the
designs generated from prompts are within the object class of application, i.e.
diverse and novel designs need to be realistic, and, second, that more research
is required to develop methods where the strength of text prompt variations and
the resulting variations of the 3D designs share causal relations to some
degree to improve the optimization.

本文研究了在工程领域中，利用深度文本到 3D 模型的潜力，将其与计算模拟设计优化中的三维资产相结合，探讨了由于自然语言的不同解释而导致优化框架的挑战，并在航空动力学优化中，成功地利用 Shap-E 和两种不同的表示文本提示的方法，实现自动进化设计优化框架的构建，研究结果表明需要确保生成的设计在应用领域的对象类别内，并需要进一步研究如何提高文本提示变化强度和 3D 设计变化的因果关系。

用于工程设计优化的大型语言和文本到 3D 模型

Large Language and Text-to-3D Models for Engineering Design Optimization

Can the latent spaces of modern generative neural rendering models serve as
representations for 3D-aware discriminative visual understanding tasks? We use
retrieval as a proxy for measuring the metric learning properties of the latent
spaces of Shap-E, including capturing view-independence and enabling the
aggregation of scene representations from the representations of individual
image views, and find that Shap-E representations outperform those of the
classical EfficientNet baseline representations zero-shot, and is still
competitive when both methods are trained using a contrative loss. These
findings give preliminary indication that 3D-based rendering and generative
models can yield useful representations for discriminative tasks in our
innately 3D-native world. Our code is available at
https://github.com/michaelwilliamtang/golden-retriever.

本研究评估了现代生成式神经渲染模型的潜在空间是否可以作为具有三维感知的区分性视觉理解任务的表示，并使用检索作为度量学习属性的代理，发现 Shap-E 表示在零样本情况下优于经典 EfficientNet 基线表示，并且在使用对比损失训练两种方法时仍然具有竞争力。