In this paper, we present our champion solution to the Global Artificial
Intelligence Technology Innovation Competition Track 1: Medical Imaging
Diagnosis Report Generation. We select CPT-BASE as our base model for the text
generation task. During the pre-training stage, we delete the mask language
modeling task of CPT-BASE and instead reconstruct the vocabulary, adopting a
span mask strategy and gradually increasing the number of masking ratios to
perform the denoising auto-encoder pre-training task. In the fine-tuning stage,
we design iterative retrieval augmentation and noise-aware similarity bucket
prompt strategies. The retrieval augmentation constructs a mini-knowledge base,
enriching the input information of the model, while the similarity bucket
further perceives the noise information within the mini-knowledge base, guiding
the model to generate higher-quality diagnostic reports based on the similarity
prompts. Surprisingly, our single model has achieved a score of 2.321 on
leaderboard A, and the multiple model fusion scores are 2.362 and 2.320 on the
A and B leaderboards respectively, securing first place in the rankings.

本研究通过在预训练和微调阶段采用不同的策略，结合知识库和相似提示来构建诊断报告，取得了在全球人工智能技术创新大赛医学影像诊断报告生成赛道中的第一名成绩。

2023 全球人工智能技术创新竞赛第一赛道冠军解决方案

First Place Solution of 2023 Global Artificial Intelligence Technology  Innovation Competition Track 1

Multimodal deep learning utilizing imaging and diagnostic reports has made
impressive progress in the field of medical imaging diagnostics, demonstrating
a particularly strong capability for auxiliary diagnosis in cases where
sufficient annotation information is lacking. Nonetheless, localizing diseases
accurately without detailed positional annotations remains a challenge.
Although existing methods have attempted to utilize local information to
achieve fine-grained semantic alignment, their capability in extracting the
fine-grained semantics of the comprehensive contextual within reports is
limited. To solve this problem, we introduce a new method that takes full
sentences from textual reports as the basic units for local semantic alignment.
Our approach combines chest X-ray images with their corresponding textual
reports, performing contrastive learning at both global and local levels. The
leading results obtained by our method on multiple datasets confirm its
efficacy in the task of lesion localization.

利用图像和诊断报告的多模态深度学习在医学影像诊断领域取得了显著进展，尤其在缺乏足够注释信息的辅助诊断方面具有强大的能力，然而，没有详细位置注释的准确定位疾病仍然是一个挑战，现有的方法已尝试利用局部信息实现细粒度语义对齐，但其在提取综合上下文内的细粒度语义能力有限，为解决这一问题，我们提出了一种新的方法，将文本报告中的完整句子作为局部语义对齐的基本单元，我们的方法结合了胸部 X 射线图像和相应的文本报告，在全局和局部层面进行对比学习，我们的方法在多个数据集上取得的领先结果证实了其在病灶定位任务中的有效性。

病变定位的多模态自监督学习

Multimodal self-supervised learning for lesion localization

The inability to interpret the model prediction in semantically and visually
meaningful ways is a well-known shortcoming of most existing computer-aided
diagnosis methods. In this paper, we propose MDNet to establish a direct
multimodal mapping between medical images and diagnostic reports that can read
images, generate diagnostic reports, retrieve images by symptom descriptions,
and visualize attention, to provide justifications of the network diagnosis
process. MDNet includes an image model and a language model. The image model is
proposed to enhance multi-scale feature ensembles and utilization efficiency.
The language model, integrated with our improved attention mechanism, aims to
read and explore discriminative image feature descriptions from reports to
learn a direct mapping from sentence words to image pixels. The overall network
is trained end-to-end by using our developed optimization strategy. Based on a
pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we
conduct sufficient experiments to demonstrate that MDNet outperforms
comparative baselines. The proposed image model obtains state-of-the-art
performance on two CIFAR datasets as well.

本文提出 MDNet，一种基于计算机视觉和自然语言处理的医学影像诊断模型，它可以自动阅读影像，生成诊断报告，并直接对症状进行图像检索，可视化诊断过程，实现端到端的优化训练，从而显著提高了在膀胱癌影像和报告数据集上的诊断准确率。