Multimodal data pervades various domains, including healthcare, social media,
and transportation, where multimodal graphs play a pivotal role. Machine
learning on multimodal graphs, referred to as multimodal graph learning (MGL),
is essential for successful artificial intelligence (AI) applications. The
burgeoning research in this field encompasses diverse graph data types and
modalities, learning techniques, and application scenarios. This survey paper
conducts a comparative analysis of existing works in multimodal graph learning,
elucidating how multimodal learning is achieved across different graph types
and exploring the characteristics of prevalent learning techniques.
Additionally, we delineate significant applications of multimodal graph
learning and offer insights into future directions in this domain.
Consequently, this paper serves as a foundational resource for researchers
seeking to comprehend existing MGL techniques and their applicability across
diverse scenarios.

多模态图学习的比较分析，阐明不同图类型的多模态学习方式，探索主要学习技术的特点，以及多模态图学习的重要应用和未来方向。

多模态图学习：一项调查

Learning on Multimodal Graphs: A Survey

Multimodal learning combines multiple data modalities, broadening the types
and complexity of data our models can utilize: for example, from plain text to
image-caption pairs. Most multimodal learning algorithms focus on modeling
simple one-to-one pairs of data from two modalities, such as image-caption
pairs, or audio-text pairs. However, in most real-world settings, entities of
different modalities interact with each other in more complex and multifaceted
ways, going beyond one-to-one mappings. We propose to represent these complex
relationships as graphs, allowing us to capture data with any number of
modalities, and with complex relationships between modalities that can flexibly
vary from one sample to another. Toward this goal, we propose Multimodal Graph
Learning (MMGL), a general and systematic framework for capturing information
from multiple multimodal neighbors with relational structures among them. In
particular, we focus on MMGL for generative tasks, building upon pretrained
Language Models (LMs), aiming to augment their text generation with multimodal
neighbor contexts. We study three research questions raised by MMGL: (1) how
can we infuse multiple neighbor information into the pretrained LMs, while
avoiding scalability issues? (2) how can we infuse the graph structure
information among multimodal neighbors into the LMs? and (3) how can we
finetune the pretrained LMs to learn from the neighbor context in a
parameter-efficient manner? We conduct extensive experiments to answer these
three questions on MMGL and analyze the empirical results to pave the way for
future MMGL research.

多模态图学习 (MMGL) 是一个通用、系统化的框架，用于捕捉具有关系结构的多模态邻居的信息并且在预训练的语言模型中增强其文本生成能力。

多模态图学习的生成任务

Multimodal Graph Learning for Generative Tasks

Artificial intelligence for graphs has achieved remarkable success in
modeling complex systems, ranging from dynamic networks in biology to
interacting particle systems in physics. However, the increasingly
heterogeneous graph datasets call for multimodal methods that can combine
different inductive biases: the set of assumptions that algorithms use to make
predictions for inputs they have not encountered during training. Learning on
multimodal datasets presents fundamental challenges because the inductive
biases can vary by data modality and graphs might not be explicitly given in
the input. To address these challenges, multimodal graph AI methods combine
different modalities while leveraging cross-modal dependencies using graphs.
Diverse datasets are combined using graphs and fed into sophisticated
multimodal architectures, specified as image-intensive, knowledge-grounded and
language-intensive models. Using this categorization, we introduce a blueprint
for multimodal graph learning, use it to study existing methods and provide
guidelines to design new models.

本文介绍了一种多模式图形学习的蓝图，该模型通过将不同模态的数据集组合成图形，并使用图形来利用跨模态的依赖关系，同时利用图形作为指导提供了图形学习方法的设计方案和指导，以解决在多模态数据集中学习的基本挑战。