Automated radiographic report generation is a challenging cross-domain task
that aims to automatically generate accurate and semantic-coherence reports to
describe medical images. Despite the recent progress in this field, there are
still many challenges at least in the following aspects. First, radiographic
images are very similar to each other, and thus it is difficult to capture the
fine-grained visual differences using CNN as the visual feature extractor like
many existing methods. Further, semantic information has been widely applied to
boost the performance of generation tasks (e.g. image captioning), but existing
methods often fail to provide effective medical semantic features. Toward
solving those problems, in this paper, we propose a memory-augmented sparse
attention block utilizing bilinear pooling to capture the higher-order
interactions between the input fine-grained image features while producing
sparse attention. Moreover, we introduce a novel Medical Concepts Generation
Network (MCGN) to predict fine-grained semantic concepts and incorporate them
into the report generation process as guidance. Our proposed method shows
promising performance on the recently released largest benchmark MIMIC-CXR. It
outperforms multiple state-of-the-art methods in image captioning and medical
report generation.

本文提出了一种利用稀疏注意力块和双线性池化捕捉细粒度图像特征之间高阶交互的记忆增强型稀疏注意力块，并引入新颖的医学概念生成网络来预测细粒度语义概念，并将其作为指导结合到自动生成医学报告的过程中，最终在 MIMIC-CXR 及图像字幕生成和医学报告生成多个领域的基准测试中，表现出了优异的性能。