Medical images are widely used in clinical practice for diagnosis.
Automatically generating interpretable medical reports can reduce radiologists'
burden and facilitate timely care. However, most existing approaches to
automatic report generation require sufficient labeled data for training. In
addition, the learned model can only generate reports for the training classes,
lacking the ability to adapt to previously unseen novel diseases. To this end,
we propose a lesion guided explainable few weak-shot medical report generation
framework that learns correlation between seen and novel classes through visual
and semantic feature alignment, aiming to generate medical reports for diseases
not observed in training. It integrates a lesion-centric feature extractor and
a Transformer-based report generation module. Concretely, the lesion-centric
feature extractor detects the abnormal regions and learns correlations between
seen and novel classes with multi-view (visual and lexical) embeddings. Then,
features of the detected regions and corresponding embeddings are concatenated
as multi-view input to the report generation module for explainable report
generation, including text descriptions and corresponding abnormal regions
detected in the images. We conduct experiments on FFA-IR, a dataset providing
explainable annotations, showing that our framework outperforms others on
report generation for novel diseases.

提出了一种病变引导的可解释性小样本医学报告生成框架，该框架通过视觉和语义特征对齐学习见过的和新疾病类型之间的相关性，从而生成未在训练中观察到的疾病的报告。