Supervised multi-modal learning involves mapping multiple modalities to a
target label. Previous studies in this field have concentrated on capturing in
isolation either the inter-modality dependencies (the relationships between
different modalities and the label) or the intra-modality dependencies (the
relationships within a single modality and the label). We argue that these
conventional approaches that rely solely on either inter- or intra-modality
dependencies may not be optimal in general. We view the multi-modal learning
problem from the lens of generative models where we consider the target as a
source of multiple modalities and the interaction between them. Towards that
end, we propose inter- & intra-modality modeling (I2M2) framework, which
captures and integrates both the inter- and intra-modality dependencies,
leading to more accurate predictions. We evaluate our approach using real-world
healthcare and vision-and-language datasets with state-of-the-art models,
demonstrating superior performance over traditional methods focusing only on
one type of modality dependency.

通过提出的 I2M2 框架，我们捕捉并整合了模态间和模态内的依赖关系，从而实现了更精确的预测。在真实世界的医疗和视觉与语言数据集上，我们使用最先进的模型评估了我们的方法，并且表现出优于传统方法的性能，传统方法仅关注一种模态依赖。