Medical vision-and-language pre-training provides a feasible solution to
extract effective vision-and-language representations from medical images and
texts. However, few studies have been dedicated to this field to facilitate
medical vision-and-language understanding. In this paper, we propose a
self-supervised learning paradigm with multi-modal masked autoencoders
(M$^3$AE), which learn cross-modal domain knowledge by reconstructing missing
pixels and tokens from randomly masked images and texts. There are three key
designs to make this simple approach work. First, considering the different
information densities of vision and language, we adopt different masking ratios
for the input image and text, where a considerably larger masking ratio is used
for images. Second, we use visual and textual features from different layers to
perform the reconstruction to deal with different levels of abstraction in
visual and language. Third, we develop different designs for vision and
language decoders (i.e., a Transformer for vision and a multi-layer perceptron
for language). To perform a comprehensive evaluation and facilitate further
research, we construct a medical vision-and-language benchmark including three
tasks. Experimental results demonstrate the effectiveness of our approach,
where state-of-the-art results are achieved on all downstream tasks. Besides,
we conduct further analysis to better verify the effectiveness of different
components of our approach and various settings of pre-training. The source
code is available at~https://github.com/zhjohnchan/M3AE.

提出了一种基于 M$^3$AE 的自监督学习模型，通过随机遮盖图像和文本中的像素和标记来学习跨模态领域知识的医学视觉语言模型，并在三个任务上实现了最先进水平的结果。