Understanding 3D medical image volumes is a critical task in the medical
domain. However, existing 3D convolution and transformer-based methods have
limited semantic understanding of an image volume and also need a large set of
volumes for training. Recent advances in multi-modal large language models
(MLLMs) provide a new and promising way to understand images with the help of
text descriptions. However, most current MLLMs are designed for 2D natural
images. To enhance the 3D medical image understanding with 2D MLLMs, we propose
a novel pre-training framework called Med3DInsight, which marries existing 3D
image encoders with 2D MLLMs and bridges them via a designed Plane-Slice-Aware
Transformer (PSAT) module. Extensive experiments demonstrate our SOTA
performance on two downstream segmentation and classification tasks, including
three public datasets with CT and MRI modalities and comparison to more than
ten baselines. Med3DInsight can be easily integrated into any current 3D
medical image understanding network and improves its performance by a good
margin.

通过结合现有的 3D 图像编码器和 2D MLLMs，并通过设计的 Plane-Slice-Aware Transformer（PSAT）模块，提出了一种名为 Med3DInsight 的新颖预训练框架，用于增强对 3D 医学图像的理解。实验证明，在两个下游分割和分类任务中，包括使用 CT 和 MRI 模式的三个公共数据集以及与十多个基准模型的比较中，Med3DInsight 取得了最先进的性能，并且可以轻松集成到任何当前的 3D 医学图像理解网络中，显著提高其性能。