Multi-modal machine learning (MMML), which involves integrating multiple
modalities of data and their corresponding processing methods, has demonstrated
promising results in various practical applications, such as text-to-image
translation. This review paper summarizes the recent progress and challenges in
using MMML for engineering design tasks. First, we introduce the different data
modalities commonly used as design representations and involved in MMML,
including text, 2D pixel data (e.g., images and sketches), and 3D shape data
(e.g., voxels, point clouds, and meshes). We then provide an overview of the
various approaches and techniques used for representing, fusing, aligning,
synthesizing, and co-learning multi-modal data as five fundamental concepts of
MMML. Next, we review the state-of-the-art capabilities of MMML that
potentially apply to engineering design tasks, including design knowledge
retrieval, design evaluation, and design synthesis. We also highlight the
potential benefits and limitations of using MMML in these contexts. Finally, we
discuss the challenges and future directions in using MMML for engineering
design, such as the need for large labeled multi-modal design datasets, robust
and scalable algorithms, integrating domain knowledge, and handling data
heterogeneity and noise. Overall, this review paper provides a comprehensive
overview of the current state and prospects of MMML for engineering design
applications.

该综述探讨了多模式机器学习在工程设计任务中的应用，介绍了用作设计表示及涉及到多模式学习的不同的数据模式，简要说明了五种多模式数据的表示和融合方法、以及其在设计知识检索、设计评估和设计综合中可能的应用，同时强调在工程设计中使用多模式机器学习时可能面临的挑战和未来方向。