Generating radiology reports automatically reduces the workload of radiologists and helps the diagnoses of specific diseases. Many existing methods take this task as modality transfer process. However, since the key information related to disease accounts for a small proportion in both image and report, it is hard for the model to learn the latent relation between the radiology image and its report, thus failing to generate fluent and accurate radiology reports. To tackle this problem, we propose a memory-based cross-modal semantic alignment model (MCSAM) following an encoder-decoder paradigm. MCSAM includes a well initialized long-term clinical memory bank to learn disease-related representations as well as prior knowledge for different modalities to retrieve and use the retrieved memory to perform feature consolidation. To ensure the semantic consistency of the retrieved cross modal prior knowledge, a cross-modal semantic alignment module (SAM) is proposed. SAM is also able to generate semantic visual feature embeddings which can be added to the decoder and benefits report generation. More importantly, to memorize the state and additional information while generating reports with the decoder, we use learnable memory tokens which can be seen as prompts. Extensive experiments demonstrate the promising performance of our proposed method which generates state-of-the-art performance on the MIMIC-CXR dataset.

提出了一种基于记忆的跨模态语义对齐模型（MCSAM），它包括一个良好初始化的长期临床记忆库来学习与疾病相关的表示，以及检索和使用检索到的记忆进行特征整合的先验知识，同时通过交叉模态语义对齐模块（SAM）生成语义视觉特征嵌入，有助于报告生成；通过使用可学习的记忆提示记住解码器在报告生成过程中的状态和附加信息，实验证明该方法在MIMIC-CXR数据集上具有有前景的性能表现。

基于记忆的跨模态语义对齐网络用于放射学报告生成