Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit---Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.

本文提出了一种基于知识推理的、融合传统方法和现代学习方法的医学报告生成方法，将医学报告生成分解为医学异常图形学习和自然语言建模两个部分，通过编码、检索和改写等子模块实现；其中，核心部分是图形变换器，可以实现知识图谱、图像和序列等多个领域的图形化高层语义转换。实验表明，该方法在两项医学报告测试中取得了最佳结果，达到最优医学异常和疾病分类准确性并提高了人工评估性能。

基于知识驱动的医学图像报告生成模型：编码、检索、改写