Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.

本研究提出一种新的混合检索-生成强化学习代理（HRGR-Agent）算法，以生成与医学图像相对应的结构化、丰富和稳健的报告，同时提高医学术语检测精度和人工评估表现。该算法通过层次决策过程来实现，使用强化学习进行更新。实验结果表明，该算法在医学报告数据集上取得了最先进的结果。

基于检索-生成增强的混合智能体用于医学图像报告生成