The adoption of large language models (LLMs) in healthcare has attracted
significant research interest. However, their performance in healthcare remains
under-investigated and potentially limited, due to i) they lack rich
domain-specific knowledge and medical reasoning skills; and ii) most
state-of-the-art LLMs are unimodal, text-only models that cannot directly
process multimodal inputs. To this end, we propose a multimodal medical
collaborative reasoning framework \textbf{MultiMedRes}, which incorporates a
learner agent to proactively gain essential information from domain-specific
expert models, to solve medical multimodal reasoning problems. Our method
includes three steps: i) \textbf{Inquire}: The learner agent first decomposes
given complex medical reasoning problems into multiple domain-specific
sub-problems; ii) \textbf{Interact}: The agent then interacts with
domain-specific expert models by repeating the ``ask-answer'' process to
progressively obtain different domain-specific knowledge; iii)
\textbf{Integrate}: The agent finally integrates all the acquired
domain-specific knowledge to accurately address the medical reasoning problem.
We validate the effectiveness of our method on the task of difference visual
question answering for X-ray images. The experiments demonstrate that our
zero-shot prediction achieves state-of-the-art performance, and even
outperforms the fully supervised methods. Besides, our approach can be
incorporated into various LLMs and multimodal LLMs to significantly boost their
performance.

我们提出了一个多模态医疗协作推理框架 MultiMedRes，通过学习者代理从领域特定专家模型获取必要信息来解决医疗多模态推理问题，并在 X 射线图像的区别视觉问答任务中验证了方法的有效性。

主动协作框架：零样本多模态医学推理的查询、交互和集成

Inquire, Interact, and Integrate: A Proactive Agent Collaborative  Framework for Zero-Shot Multimodal Medical Reasoning

Difference visual question answering (diff-VQA) is a challenging task that
requires answering complex questions based on differences between a pair of
images. This task is particularly important in reading chest X-ray images
because radiologists often compare multiple images of the same patient taken at
different times to track disease progression and changes in its severity in
their clinical practice. However, previous works focused on designing specific
network architectures for the diff-VQA task, missing opportunities to enhance
the model's performance using a pretrained vision-language model (VLM). Here,
we introduce a novel VLM called PLURAL, which is pretrained on natural and
longitudinal chest X-ray data for the diff-VQA task. The model is developed
using a step-by-step approach, starting with being pretrained on natural images
and texts, followed by being trained using longitudinal chest X-ray data. The
longitudinal data consist of pairs of X-ray images, along with question-answer
sets and radiologist's reports that describe the changes in lung abnormalities
and diseases over time. Our experimental results show that the PLURAL model
outperforms state-of-the-art methods not only in diff-VQA for longitudinal
X-rays but also in conventional VQA for a single X-ray image. Through extensive
experiments, we demonstrate the effectiveness of the proposed VLM architecture
and pretraining method in improving the model's performance.

PLURAL 模型是一种针对差异化视觉问答任务的预训练视觉 - 语言模型，经过实验证明在研究中能够提高模型的性能。