To achieve high-quality results, diffusion models must be trained on large datasets. This can be notably prohibitive for models in specialized domains, such as computational pathology. Conditioning on labeled data is known to help in data-efficient model training. Therefore, histopathology reports, which are rich in valuable clinical information, are an ideal choice as guidance for a histopathology generative model. In this paper, we introduce PathLDM, the first text-conditioned Latent Diffusion Model tailored for generating high-quality histopathology images. Leveraging the rich contextual information provided by pathology text reports, our approach fuses image and textual data to enhance the generation process. By utilizing GPT's capabilities to distill and summarize complex text reports, we establish an effective conditioning mechanism. Through strategic conditioning and necessary architectural enhancements, we achieved a SoTA FID score of 7.64 for text-to-image generation on the TCGA-BRCA dataset, significantly outperforming the closest text-conditioned competitor with FID 30.1.

本研究介绍了PathLDM，这是一种专为生成高质量组织病理学图像而设计的首个文本条件潜在扩散模型。通过利用病理学文本报告所提供的丰富语境信息，结合图像和文本数据，通过GPT的能力进行复杂文本报告的提炼和总结，我们实现了在TCGA-BRCA数据集上文本到图像生成方面的SoTA FID得分为7.64，显著优于最接近的文本条件竞争对手的30.1的FID得分。

PathLDM：用于组织病理学的文本条件潜变扩散模型