As advances in large language models (LLMs) and multimodal techniques
continue to mature, the development of general-purpose multimodal large
language models (MLLMs) has surged, with significant applications in natural
image interpretation. However, the field of pathology has largely remained
untapped in this regard, despite the growing need for accurate, timely, and
personalized diagnostics. To bridge the gap in pathology MLLMs, we present the
PathAsst in this study, which is a generative foundation AI assistant to
revolutionize diagnostic and predictive analytics in pathology. To develop
PathAsst, we collect over 142K high-quality pathology image-text pairs from a
variety of reliable sources, including PubMed, comprehensive pathology
textbooks, reputable pathology websites, and private data annotated by
pathologists. Leveraging the advanced capabilities of ChatGPT/GPT-4, we
generate over 180K instruction-following samples. Furthermore, we devise
additional instruction-following data, specifically tailored for the invocation
of the pathology-specific models, allowing the PathAsst to effectively interact
with these models based on the input image and user intent, consequently
enhancing the model's diagnostic capabilities. Subsequently, our PathAsst is
trained based on Vicuna-13B language model in coordination with the CLIP vision
encoder. The results of PathAsst show the potential of harnessing the
AI-powered generative foundation model to improve pathology diagnosis and
treatment processes. We are committed to open-sourcing our meticulously curated
dataset, as well as a comprehensive toolkit designed to aid researchers in the
extensive collection and preprocessing of their own datasets. Resources can be
obtained at
this https URL

本文提出了 PathAsst，一种生成式 AI 助手，利用了 ChatGPT/GPT-4 和 Vicuna-13B 语言模型与 CLIP 视觉编码器，对 142K 高质量病理图像文本对进行了训练。结果表明，利用这种 AI 模型可以改善病理诊断和治疗过程。