Multimodal large-scale pretraining has shown impressive performance gains for
unstructured data including language, image, audio, and video. Yet, the
scenario most prominent in real-world applications is the existence of
combination of structured (including tabular and time-series) and unstructured
data, and this has so far been understudied. Towards this end, we propose
LANISTR, a novel attention-based framework to learn from LANguage, Image, and
STRuctured data. We introduce a new multimodal fusion module with a
similarity-based multimodal masking loss that enables LANISTR to learn
cross-modal relations from large-scale multimodal data with missing modalities
during training and test time. On two publicly available challenging datasets,
MIMIC-IV and Amazon Product Review, LANISTR achieves absolute improvements of
6.47% (AUROC) and up to 17.69% (accuracy), respectively, compared to the
state-of-the-art multimodal models while showing superior generalization
capabilities.

LANISTR 是一种基于注意力机制的框架，能够学习语言、图像和结构化数据，并通过一种基于相似度的多模态掩蔽损失函数来学习大规模多模态数据中跨模态的关系。在两个具有挑战性的公开数据集 MIMIC-IV 和 Amazon 产品评论上，与现有的多模态模型相比，LANISTR 的绝对改进分别为 6.47％（AUROC）和高达 17.69％（准确性），同时显示出更出色的泛化能力。

LANISTR：来自结构化和非结构化数据的多模态学习

LANISTR: Multimodal Learning from Structured and Unstructured Data

Modern Review Helpfulness Prediction systems are dependent upon multiple
modalities, typically texts and images. Unfortunately, those contemporary
approaches pay scarce attention to polish representations of cross-modal
relations and tend to suffer from inferior optimization. This might cause harm
to model's predictions in numerous cases. To overcome the aforementioned
issues, we propose Multimodal Contrastive Learning for Multimodal Review
Helpfulness Prediction (MRHP) problem, concentrating on mutual information
between input modalities to explicitly elaborate cross-modal relations. In
addition, we introduce Adaptive Weighting scheme for our contrastive learning
approach in order to increase flexibility in optimization. Lastly, we propose
Multimodal Interaction module to address the unalignment nature of multimodal
data, thereby assisting the model in producing more reasonable multimodal
representations. Experimental results show that our method outperforms prior
baselines and achieves state-of-the-art results on two publicly available
benchmark datasets for MRHP problem.

本研究旨在提高现代评论有用性预测系统的性能，提出了一种基于多模态对比学习的方法，聚焦于输入模态之间的信息，提高了跨模态关系的显式表达，并引入自适应加权策略以提高优化的灵活性，同时提供了多模态交互模块以解决多模态数据的分布不一致问题，实验证明该方法优于基线算法，在两个公开数据集上实现了最先进的结果。