This paper presents a Tri-branch Neural Fusion (TNF) approach designed for
classifying multimodal medical images and tabular data. It also introduces two
solutions to address the challenge of label inconsistency in multimodal
classification. Traditional methods in multi-modality medical data
classification often rely on single-label approaches, typically merging
features from two distinct input modalities. This becomes problematic when
features are mutually exclusive or labels differ across modalities, leading to
reduced accuracy. To overcome this, our TNF approach implements a tri-branch
framework that manages three separate outputs: one for image modality, another
for tabular modality, and a third hybrid output that fuses both image and
tabular data. The final decision is made through an ensemble method that
integrates likelihoods from all three branches. We validate the effectiveness
of TNF through extensive experiments, which illustrate its superiority over
traditional fusion and ensemble methods in various convolutional neural
networks and transformer-based architectures across multiple datasets.

本文介绍了一种用于分类多模态医学图像和表格数据的三分支神经融合（TNF）方法，并提出了两种解决多模态分类中标签不一致性挑战的解决方案。通过在三个分支中管理图像模态、表格模态和融合数据的混合输出，最终的决策是通过集成方法从三个分支的可能性中得出的。通过广泛的实验验证了 TNF 方法在各种卷积神经网络和基于 Transformer 的架构以及多个数据集上的优越性。