In this study, we introduce a multi-modal approach that efficiently
integrates multi-scale clinical and dermoscopy features within a single
network, thereby substantially reducing model parameters. The proposed method
includes three novel fusion schemes.
Firstly, unlike current methods that usually employ two individual models for
for clinical and dermoscopy modalities, we verified that multimodal feature can
be learned by sharing the parameters of encoder while leaving the individual
modal-specific classifiers.
Secondly, the shared cross-attention module can replace the individual one to
efficiently interact between two modalities at multiple layers.
Thirdly, different from current methods that equally optimize dermoscopy and
clinical branches, inspired by prior knowledge that dermoscopy images play a
more significant role than clinical images, we propose a novel biased loss.
This loss guides the single-shared network to prioritize dermoscopy information
over clinical information, implicitly learning a better joint feature
representation for the modal-specific task.
Extensive experiments on a well-recognized Seven-Point Checklist (SPC)
dataset and a collected dataset demonstrate the effectiveness of our method on
both CNN and Transformer structures. Furthermore, our method exhibits
superiority in both accuracy and model parameters compared to currently
advanced methods.

我们提出了一种多模态方法，通过单一网络高效地整合多尺度临床和皮肤镜特征，从而大幅减少模型参数。该方法包括三种新颖的融合方案，通过共享编码器参数并保留各自的模态专属分类器、使用共享的交叉注意力模块多层次地互动两种模态、以及通过引入偏置损失将皮肤镜信息优先于临床信息，隐式地学习更好的联合特征表示。在一个公认的关键点核查表（SPC）数据集和一个采集的数据集上进行的大量实验证明了我们方法在 CNN 和 Transformer 结构上的有效性。此外，与目前先进的方法相比，我们的方法在准确性和模型参数方面都表现出优势。