The gigapixel scale of whole slide images (WSIs) poses a challenge for
histopathology multi-modal chatbots, requiring a global WSI analysis for
diagnosis, compounding evidence from different WSI patches. Current visual
instruction datasets, generated through large language models, focus on
creating question/answer pairs for individual image patches, which may lack
diagnostic capacity on their own in histopathology, further complicated by the
absence of spatial grounding in histopathology image captions. To bridge this
gap, we introduce Quilt-Instruct, a large-scale dataset of 107,131
histopathology-specific instruction question/answer pairs, that is collected by
leveraging educational histopathology videos from YouTube, which provides
spatial localization of captions by automatically extracting narrators' cursor
movements. In addition, we provide contextual reasoning by extracting diagnosis
and supporting facts from the entire video content to guide the extrapolative
reasoning of GPT-4. Using Quilt-Instruct, we train Quilt-LLaVA, which can
reason beyond the given single image patch, enabling diagnostic reasoning and
the capability of spatial awareness. To evaluate Quilt-LLaVA, we propose a
comprehensive evaluation dataset created from 985 images and 1283
human-generated question-answers. We also thoroughly evaluate Quilt-LLaVA using
public histopathology datasets, where Quilt-LLaVA significantly outperforms
SOTA by over 10% on relative GPT-4 score and 4% and 9% on open and closed set
VQA. Our code, data, and model are publicly available at quilt-llava.github.io.

通过 Quilt-Instruct 数据集和 Quilt-LLaVA 模型，本研究提出了一种能够在全幅切片图像中进行诊断推理和空间感知的多模态组织病理学问答系统。该模型在多个公共组织病理数据集上表现出比 SOTA 模型高出超过 10% 的性能。