A lot of deep learning (DL) research these days is mainly focused on
improving on quantitative metrics regardless of other factors. In human
centered applications, like skin lesion classification in dermatology,
DL-driven clinical decision support systems are still in their infancy due to
the limited transparency of their decision-making process. Moreover, the lack
of procedures that can explain the behavior of trained DL algorithms leads to
almost no trust from the clinical physicians. To diagnose skin lesions,
dermatologists rely on both visual assessment of the disease and the data
gathered from the anamnesis of the patient. Data-driven algorithms dealing with
multi-modal data are limited by the separation of feature-level and
decision-level fusion procedures required by convolutional architectures. To
address this issue, we enable single-stage multi-modal data fusion via the
attention mechanism of transformer-based architectures to aid in the diagnosis
of skin diseases. Our method beats other state-of-the-art single- and
multi-modal DL architectures in both image rich and patient-data rich
environments. Additionally, the choice of the architecture enables native
interpretability support for the classification task both in image and metadata
domain with no additional modifications necessary.

本研究利用 transformer-based 架构实现了单阶段多模态数据融合，并在图像与病患数据丰富的环境下击败了其他单模态和多模态深度学习架构，同时，所选架构还提供了本地可解释性支持。