Whole slide images are the foundation of digital pathology for the diagnosis
and treatment of carcinomas. Writing pathology reports is laborious and
error-prone for inexperienced pathologists. To reduce the workload and improve
clinical automation, we investigate how to generate pathology reports given
whole slide images. On the data end, we curated the largest WSI-text dataset
(TCGA-PathoText). In specific, we collected nearly 10000 high-quality WSI-text
pairs for visual-language models by recognizing and cleaning pathology reports
which narrate diagnostic slides in TCGA. On the model end, we propose the
multiple instance generative model (MI-Gen) which can produce pathology reports
for gigapixel WSIs. We benchmark our model on the largest subset of
TCGA-PathoText. Experimental results show our model can generate pathology
reports which contain multiple clinical clues. Furthermore, WSI-text prediction
can be seen as an approach of visual-language pre-training, which enables our
model to be transferred to downstream diagnostic tasks like carcinoma grading
and phenotyping. We observe that simple semantic extraction from the pathology
reports can achieve the best performance (0.838 of F1 score) on BRCA subtyping
without adding extra parameters or tricky fine-tuning. Our collected dataset
and related code will all be publicly available.

使用全幻灯片图像生成病理报告的多示例生成模型 (MI-Gen) 可减轻经验不足的病理医生的工作负担并提高临床自动化，同时简单的语义提取可以实现最佳性能且公开可用。

MI-Gen：用于千兆像素全切片图像的病理报告多示例生成

MI-Gen: Multiple Instance Generation of Pathology Reports for Gigapixel  Whole-Slide Images

Medical imaging is frequently used in clinical practice and trials for
diagnosis and treatment. Writing imaging reports is time-consuming and can be
error-prone for inexperienced radiologists. Therefore, automatically generating
radiology reports is highly desired to lighten the workload of radiologists and
accordingly promote clinical automation, which is an essential task to apply
artificial intelligence to the medical domain. In this paper, we propose to
generate radiology reports with memory-driven Transformer, where a relational
memory is designed to record key information of the generation process and a
memory-driven conditional layer normalization is applied to incorporating the
memory into the decoder of Transformer. Experimental results on two prevailing
radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed
approach outperforms previous models with respect to both language generation
metrics and clinical evaluations. Particularly, this is the first work
reporting the generation results on MIMIC-CXR to the best of our knowledge.
Further analyses also demonstrate that our approach is able to generate long
reports with necessary medical terms as well as meaningful image-text attention
mappings.

本文提出了使用记忆驱动 Transformer 来生成放射学报告，实验证明该方法可以以更高的质量、更长的长度和更多的医学术语产生放射学报告，此为我们所知第一次在 MIMIC-CXR 上实现放射学报告的生成结果。