The missing modality issue is critical but non-trivial to be solved by
multi-modal models. Current methods aiming to handle the missing modality
problem in multi-modal tasks, either deal with missing modalities only during
evaluation or train separate models to handle specific missing modality
settings. In addition, these models are designed for specific tasks, so for
example, classification models are not easily adapted to segmentation tasks and
vice versa. In this paper, we propose the Shared-Specific Feature Modelling
(ShaSpec) method that is considerably simpler and more effective than competing
approaches that address the issues above. ShaSpec is designed to take advantage
of all available input modalities during training and evaluation by learning
shared and specific features to better represent the input data. This is
achieved from a strategy that relies on auxiliary tasks based on distribution
alignment and domain classification, in addition to a residual feature fusion
procedure. Also, the design simplicity of ShaSpec enables its easy adaptation
to multiple tasks, such as classification and segmentation. Experiments are
conducted on both medical image segmentation and computer vision
classification, with results indicating that ShaSpec outperforms competing
methods by a large margin. For instance, on BraTS2018, ShaSpec improves the
SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole
tumour.

本文提出了一种叫做 Shared-Specific Feature Modelling (ShaSpec) 的方法，旨在解决多模态任务中遗漏模态问题，并通过学习共享和特定特征来更好地表示输入数据。实验结果表明，ShaSpec 在医学图像分割和计算机视觉分类等多个任务中比竞争方法表现更好。