The impression is crucial for the referring physicians to grasp key
information since it is concluded from the findings and reasoning of
radiologists. To alleviate the workload of radiologists and reduce repetitive
human labor in impression writing, many researchers have focused on automatic
impression generation. However, recent works on this task mainly summarize the
corresponding findings and pay less attention to the radiology images. In
clinical, radiographs can provide more detailed valuable observations to
enhance radiologists' impression writing, especially for complicated cases.
Besides, each sentence in findings usually focuses on single anatomy, so they
only need to be matched to corresponding anatomical regions instead of the
whole image, which is beneficial for textual and visual features alignment.
Therefore, we propose a novel anatomy-enhanced multimodal model to promote
impression generation. In detail, we first construct a set of rules to extract
anatomies and put these prompts into each sentence to highlight anatomy
characteristics. Then, two separate encoders are applied to extract features
from the radiograph and findings. Afterward, we utilize a contrastive learning
module to align these two representations at the overall level and use a
co-attention to fuse them at the sentence level with the help of
anatomy-enhanced sentence representation. Finally, the decoder takes the fused
information as the input to generate impressions. The experimental results on
two benchmark datasets confirm the effectiveness of the proposed method, which
achieves state-of-the-art results.

本文提出了一种新的解决方案来增强放射学家的结论写作，重点在于，将放射学图像和检查结果组合在一起生成结论，主要包括解剖学增强多模态模型和对比学习模块，经过两个基准数据集的实验证明，该方法显著优于其他自动结论生成模型。

利用放射学和解剖提示改善放射学摘要

Improving Radiology Summarization with Radiograph and Anatomy Prompts

The impression section of a radiology report summarizes the most prominent
observation from the findings section and is the most important section for
radiologists to communicate to physicians. Summarizing findings is
time-consuming and can be prone to error for inexperienced radiologists, and
thus automatic impression generation has attracted substantial attention. With
the encoder-decoder framework, most previous studies explore incorporating
extra knowledge (e.g., static pre-defined clinical ontologies or extra
background information). Yet, they encode such knowledge by a separate encoder
to treat it as an extra input to their models, which is limited in leveraging
their relations with the original findings. To address the limitation, we
propose a unified framework for exploiting both extra knowledge and the
original findings in an integrated way so that the critical information (i.e.,
key words and their relations) can be extracted in an appropriate way to
facilitate impression generation. In detail, for each input findings, it is
encoded by a text encoder, and a graph is constructed through its entities and
dependency tree. Then, a graph encoder (e.g., graph neural networks (GNNs)) is
adopted to model relation information in the constructed graph. Finally, to
emphasize the key words in the findings, contrastive learning is introduced to
map positive samples (constructed by masking non-key words) closer and push
apart negative ones (constructed by masking key words). The experimental
results on OpenI and MIMIC-CXR confirm the effectiveness of our proposed
method.

提出一种基于 Encoder-Decoder 框架且同时利用附加知识和原始结果的方法，利用图神经网络处理视频关系信息，采用对比学习方法强化关键词在结果中的表述，以提高放射学报告印象部分的准确性。