Existing X-ray based pre-trained vision models are usually conducted on a
relatively small-scale dataset (less than 500k samples) with limited resolution
(e.g., 224 $\times$ 224). However, the key to the success of self-supervised
pre-training large models lies in massive training data, and maintaining high
resolution in the field of X-ray images is the guarantee of effective solutions
to difficult miscellaneous diseases. In this paper, we address these issues by
proposing the first high-definition (1280 $\times$ 1280) X-ray based
pre-trained foundation vision model on our newly collected large-scale dataset
which contains more than 1 million X-ray images. Our model follows the masked
auto-encoder framework which takes the tokens after mask processing (with a
high rate) is used as input, and the masked image patches are reconstructed by
the Transformer encoder-decoder network. More importantly, we introduce a novel
context-aware masking strategy that utilizes the chest contour as a boundary
for adaptive masking operations. We validate the effectiveness of our model on
two downstream tasks, including X-ray report generation and disease
recognition. Extensive experiments demonstrate that our pre-trained medical
foundation vision model achieves comparable or even new state-of-the-art
performance on downstream benchmark datasets. The source code and pre-trained
models of this paper will be released on
this https URL

本文提出了首个高清（1280×1280）X 射线预训练视觉模型，采用了大规模数据集和自动编码器框架，引入了一种新颖的上下文感知遮蔽策略，验证了模型在 X 射线报告生成和疾病识别等下游任务上的有效性。