Automatic medical image report generation has drawn growing attention due to
its potential to alleviate radiologists' workload. Existing work on report
generation often trains encoder-decoder networks to generate complete reports.
However, such models are affected by data bias (e.g.~label imbalance) and face
common issues inherent in text generation models (e.g.~repetition). In this
work, we focus on reporting abnormal findings on radiology images; instead of
training on complete radiology reports, we propose a method to identify
abnormal findings from the reports in addition to grouping them with
unsupervised clustering and minimal rules. We formulate the task as cross-modal
retrieval and propose Conditional Visual-Semantic Embeddings to align images
and fine-grained abnormal findings in a joint embedding space. We demonstrate
that our method is able to retrieve abnormal findings and outperforms existing
generation models on both clinical correctness and text generation metrics.

该论文提出了一种基于交叉模态检索的自动医学图像报告生成方法，该方法可以从报告中识别异常发现，并将它们与无监督聚类和最小规则分组，通过使用提出的有条件构建的视觉 - 语义嵌入进行图像和小粒度异常发现对齐，其能够更好地检索异常发现并在临床正确性和文本生成度量方面优于现有的生成模型。