Automatic radiology report generation is booming due to its huge application
potential for the healthcare industry. However, existing computer vision and
natural language processing approaches to tackle this problem are limited in
two aspects. First, when extracting image features, most of them neglect
multi-view reasoning in vision and model single-view structure of medical
images, such as space-view or channel-view. However, clinicians rely on
multi-view imaging information for comprehensive judgment in daily clinical
diagnosis. Second, when generating reports, they overlook context reasoning
with multi-modal information and focus on pure textual optimization utilizing
retrieval-based methods. We aim to address these two issues by proposing a
model that better simulates clinicians' perspectives and generates more
accurate reports. Given the above limitation in feature extraction, we propose
a Globally-intensive Attention (GIA) module in the medical image encoder to
simulate and integrate multi-view vision perception. GIA aims to learn three
types of vision perception: depth view, space view, and pixel view. On the
other hand, to address the above problem in report generation, we explore how
to involve multi-modal signals to generate precisely matched reports, i.e., how
to integrate previously predicted words with region-aware visual content in
next word prediction. Specifically, we design a Visual Knowledge-guided Decoder
(VKGD), which can adaptively consider how much the model needs to rely on
visual information and previously predicted text to assist next word
prediction. Hence, our final Intensive Vision-guided Network (IVGN) framework
includes a GIA-guided Visual Encoder and the VKGD. Experiments on two
commonly-used datasets IU X-Ray and MIMIC-CXR demonstrate the superior ability
of our method compared with other state-of-the-art approaches.

通过提出具有全局密集注意力（GIA）模块的医学图像编码器和可视化知识引导解码器（VKGD），我们的研究旨在解决自动放射学报告生成中的多视角推理和多模态信息处理的问题，实现更准确的报告生成。在实验中，我们的方法在 IU X-Ray 和 MIMIC-CXR 两个常用数据集上展现出比其他最先进方法更优越的能力。