A lot of deep learning (DL) research these days is mainly focused on improving on quantitative metrics regardless of other factors. In human centered applications, like skin lesion classification in dermatology, DL-driven clinical decision support systems are still in their infancy due to the limited transparency of their decision-making process. Moreover, the lack of procedures that can explain the behavior of trained DL algorithms leads to almost no trust from the clinical physicians. To diagnose skin lesions, dermatologists rely on both visual assessment of the disease and the data gathered from the anamnesis of the patient. Data-driven algorithms dealing with multi-modal data are limited by the separation of feature-level and decision-level fusion procedures required by convolutional architectures. To address this issue, we enable single-stage multi-modal data fusion via the attention mechanism of transformer-based architectures to aid in the diagnosis of skin diseases. Our method beats other state-of-the-art single- and multi-modal DL architectures in both image rich and patient-data rich environments. Additionally, the choice of the architecture enables native interpretability support for the classification task both in image and metadata domain with no additional modifications necessary.

本研究利用transformer-based架构实现了单阶段多模态数据融合，并在图像与病患数据丰富的环境下击败了其他单模态和多模态深度学习架构，同时，所选架构还提供了本地可解释性支持。

基于Transformer的可解释多模态数据融合在皮肤病变分类中的应用