Medical imaging plays a significant role in clinical practice of medical
diagnosis, where the text reports of the images are essential in understanding
them and facilitating later treatments. By generating the reports
automatically, it is beneficial to help lighten the burden of radiologists and
significantly promote clinical automation, which already attracts much
attention in applying artificial intelligence to medical domain. Previous
studies mainly follow the encoder-decoder paradigm and focus on the aspect of
text generation, with few studies considering the importance of cross-modal
mappings and explicitly exploit such mappings to facilitate radiology report
generation. In this paper, we propose a cross-modal memory networks (CMN) to
enhance the encoder-decoder framework for radiology report generation, where a
shared memory is designed to record the alignment between images and texts so
as to facilitate the interaction and generation across modalities. Experimental
results illustrate the effectiveness of our proposed model, where
state-of-the-art performance is achieved on two widely used benchmark datasets,
i.e., IU X-Ray and MIMIC-CXR. Further analyses also prove that our model is
able to better align information from radiology images and texts so as to help
generating more accurate reports in terms of clinical indicators.

本文提出了一种跨模式记忆网络 (CMN) 来增强医学影像学中放射学报告自动生成的编码器 - 解码器框架，实验结果表明该模型能够更好地对齐放射图像和文本信息，具有更高的临床指标准确性。