The emergence of multi-modal deep learning models has made significant
impacts on clinical applications in the last decade. However, the majority of
models are limited to single-tasking, without considering disease diagnosis is
indeed a multi-task procedure. Here, we demonstrate a unified transformer model
specifically designed for multi-modal clinical tasks by incorporating
customized instruction tuning. We first compose a multi-task training dataset
comprising 13.4 million instruction and ground-truth pairs (with approximately
one million radiographs) for the customized tuning, involving both image- and
pixel-level tasks. Thus, we can unify the various vision-intensive tasks in a
single training framework with homogeneous model inputs and outputs to increase
clinical interpretability in one reading. Finally, we demonstrate the overall
superior performance of our model compared to prior arts on various chest X-ray
benchmarks across multi-tasks in both direct inference and finetuning settings.
Three radiologists further evaluate the generated reports against the recorded
ones, which also exhibit the enhanced explainability of our multi-task model.

基于多模态深度学习模型的临床应用中的多任务程序，我们提出了一个统一的变压器模型，通过整合定制化指令调整，将不同的视觉密集型任务统一在一个训练框架中，以增加临床解释性和在不同胸部 X 光基准测试中表现出的优越性能。