Recently, medical report generation, which aims to automatically generate a
long and coherent descriptive paragraph of a given medical image, has received
growing research interests. Different from the general image captioning tasks,
medical report generation is more challenging for data-driven neural models.
This is mainly due to 1) the serious data bias: the normal visual regions
dominate the dataset over the abnormal visual regions, and 2) the very long
sequence. To alleviate above two problems, we propose an AlignTransformer
framework, which includes the Align Hierarchical Attention (AHA) and the
Multi-Grained Transformer (MGT) modules: 1) AHA module first predicts the
disease tags from the input image and then learns the multi-grained visual
features by hierarchically aligning the visual regions and disease tags. The
acquired disease-grounded visual features can better represent the abnormal
regions of the input image, which could alleviate data bias problem; 2) MGT
module effectively uses the multi-grained features and Transformer framework to
generate the long medical report. The experiments on the public IU-Xray and
MIMIC-CXR datasets show that the AlignTransformer can achieve results
competitive with state-of-the-art methods on the two datasets. Moreover, the
human evaluation conducted by professional radiologists further proves the
effectiveness of our approach.

该研究提出了一个包含 Align Hierarchical Attention (AHA) 和 Multi-Grained Transformer (MGT) modules 在内的 AlignTransformer 框架，可以用于医疗图像的自动报告生成，实验表明该方法可以解决数据偏差问题，并取得了与现有最佳方法相竞争的结果。

AlignTransformer: 面向医疗报告生成的视觉区域和疾病标签的层次对齐

AlignTransformer: Hierarchical Alignment of Visual Regions and Disease  Tags for Medical Report Generation

Automatic radiology report generation is critical in clinics which can
relieve experienced radiologists from the heavy workload and remind
inexperienced radiologists of misdiagnosis or missed diagnose. Existing
approaches mainly formulate radiology report generation as an image captioning
task and adopt the encoder-decoder framework. However, in the medical domain,
such pure data-driven approaches suffer from the following problems: 1) visual
and textual bias problem; 2) lack of expert knowledge. In this paper, we
propose a knowledge-enhanced radiology report generation approach introduces
two types of medical knowledge: 1) General knowledge, which is input
independent and provides the broad knowledge for report generation; 2) Specific
knowledge, which is input dependent and provides the fine-grained knowledge for
report generation. To fully utilize both the general and specific knowledge, we
also propose a knowledge-enhanced multi-head attention mechanism. By merging
the visual features of the radiology image with general knowledge and specific
knowledge, the proposed model can improve the quality of generated reports.
Experimental results on two publicly available datasets IU-Xray and MIMIC-CXR
show that the proposed knowledge enhanced approach outperforms state-of-the-art
image captioning based methods. Ablation studies also demonstrate that both
general and specific knowledge can help to improve the performance of radiology
report generation.

本文提出一种基于多头注意力机制和引入一般和特定的医疗知识的放射学报告生成方法，结合放射学影像的视觉特征，可以提高放射学报告的质量，并在公开数据集上进行的实验结果表明，提出的知识增强方法优于最先进的基于图像字幕的方法。