Generating medical reports for X-ray images is a challenging task, particularly in an unpaired scenario where paired image-report data is unavailable for training. To address this challenge, we propose a novel model that leverages the available information in two distinct datasets, one comprising reports and the other consisting of images. The core idea of our model revolves around the notion that combining auto-encoding report generation with multi-modal (report-image) alignment can offer a solution. However, the challenge persists regarding how to achieve this alignment when pair correspondence is absent. Our proposed solution involves the use of auxiliary tasks, particularly contrastive learning and classification, to position related images and reports in close proximity to each other. This approach differs from previous methods that rely on pre-processing steps using external information stored in a knowledge graph. Our model, named MedRAT, surpasses previous state-of-the-art methods, demonstrating the feasibility of generating comprehensive medical reports without the need for paired data or external tools.

在没有成对图像-报告数据进行训练的不成对情景中，我们提出了一个新模型，通过利用两个不同数据集中的可用信息，一个由报告组成，另一个由图像组成，结合自编码报告生成和多模态（报告-图像）对齐的思路来解决这一挑战。我们的解决方案涉及使用辅助任务，特别是对比学习和分类，将相关图像和报告定位在彼此紧密相邻的位置上。与依赖预处理步骤使用知识图存储的外部信息的先前方法不同，我们的模型名为MedRAT，在不需要成对数据或外部工具的情况下，超越了先前的最先进方法，证明了生成全面医学报告的可行性。

MedRAT：通过辅助任务进行非配对式医学报告生成