Automatic generation of ophthalmic reports using data-driven neural networks
has great potential in clinical practice. When writing a report,
ophthalmologists make inferences with prior clinical knowledge. This knowledge
has been neglected in prior medical report generation methods. To endow models
with the capability of incorporating expert knowledge, we propose a Cross-modal
clinical Graph Transformer (CGT) for ophthalmic report generation (ORG), in
which clinical relation triples are injected into the visual features as prior
knowledge to drive the decoding procedure. However, two major common Knowledge
Noise (KN) issues may affect models' effectiveness. 1) Existing general
biomedical knowledge bases such as the UMLS may not align meaningfully to the
specific context and language of the report, limiting their utility for
knowledge injection. 2) Incorporating too much knowledge may divert the visual
features from their correct meaning. To overcome these limitations, we design
an automatic information extraction scheme based on natural language processing
to obtain clinical entities and relations directly from in-domain training
reports. Given a set of ophthalmic images, our CGT first restores a sub-graph
from the clinical graph and injects the restored triples into visual features.
Then visible matrix is employed during the encoding procedure to limit the
impact of knowledge. Finally, reports are predicted by the encoded cross-modal
features via a Transformer decoder. Extensive experiments on the large-scale
FFA-IR benchmark demonstrate that the proposed CGT is able to outperform
previous benchmark methods and achieve state-of-the-art performances.

该论文提出了一种用于眼科报告生成的交叉模态临床图变换器（CGT）模型，该模型利用数据驱动的神经网络和临床关系三元组注入到视觉特征中，以提高模型的效果并克服基于通用生物医学知识库的知识注入的局限性。通过自然语言处理，该模型从领域内训练报告中提取临床实体和关系数据，并在编码过程中仅限制可见矩阵的影响，实现了优于之前基准模型的最佳表现。