Conventional machine learning methods are predominantly designed to predict
outcomes based on a single data type. However, practical applications may
encompass data of diverse types, such as text, images, and audio. We introduce
interpretable tensor fusion (InTense), a multimodal learning method for
training neural networks to simultaneously learn multimodal data
representations and their interpretable fusion. InTense can separately capture
both linear combinations and multiplicative interactions of diverse data types,
thereby disentangling higher-order interactions from the individual effects of
each modality. InTense provides interpretability out of the box by assigning
relevance scores to modalities and their associations. The approach is
theoretically grounded and yields meaningful relevance scores on multiple
synthetic and real-world datasets. Experiments on six real-world datasets show
that InTense outperforms existing state-of-the-art multimodal interpretable
approaches in terms of accuracy and interpretability.

传统机器学习方法主要用于基于单一数据类型进行预测，然而实际应用可能涉及各种类型的数据，如文本、图像和音频。我们引入了可解释的张量融合（InTense），这是一种多模态学习方法，用于训练神经网络同时学习多模态数据表示和其可解释的融合。InTense 能够分别捕捉不同数据类型的线性组合和乘法相互作用，从而解开高阶相互作用和每种模态的单独影响。InTense 通过给予模态和它们的关联以相关性得分，从而提供了可解释性。该方法在理论上有基础，并且在多个合成和真实数据集上得到了有意义的相关性得分。在六个真实世界数据集上的实验证明，InTense 在准确性和可解释性方面均优于现有的多模态可解释方法。