Contrastive visual language pretraining has emerged as a powerful method for
either training new language-aware image encoders or augmenting existing
pretrained models with zero-shot visual recognition capabilities. However,
existing works typically train on large datasets of image-text pairs and have
been designed to perform downstream tasks involving only small to medium
sized-images, neither of which are applicable to the emerging field of
computational pathology where there are limited publicly available paired
image-text datasets and each image can span up to 100,000 x 100,000 pixels. In
this paper we present MI-Zero, a simple and intuitive framework for unleashing
the zero-shot transfer capabilities of contrastively aligned image and text
models on gigapixel histopathology whole slide images, enabling multiple
downstream diagnostic tasks to be carried out by pretrained encoders without
requiring any additional labels. MI-Zero reformulates zero-shot transfer under
the framework of multiple instance learning to overcome the computational
challenge of inference on extremely large images. We used over 550k pathology
reports and other available in-domain text corpora to pre-train our text
encoder. By effectively leveraging strong pre-trained encoders, our best model
pretrained on over 33k histopathology image-caption pairs achieves an average
median zero-shot accuracy of 70.2% across three different real-world cancer
subtyping tasks. Our code is available at:
this https URL

提出了 MI-Zero 框架，它将对比可视语言预训练模型的零射频转换重构为多实例学习，以在吉盘组织学全幻灯片图像上释放其零射频转换能力，从而在预先训练的编码器不需要任何额外的标签的情况下进行多个下游诊断任务。