The emergence of multi-modal deep learning models has made significant
impacts on clinical applications in the last decade. However, the majority of
models are limited to single-tasking, without considering disease diagnosis is
indeed a multi-task procedure. Here, we demonstrate a unified transformer model
specifically designed for multi-modal clinical tasks by incorporating
customized instruction tuning. We first compose a multi-task training dataset
comprising 13.4 million instruction and ground-truth pairs (with approximately
one million radiographs) for the customized tuning, involving both image- and
pixel-level tasks. Thus, we can unify the various vision-intensive tasks in a
single training framework with homogeneous model inputs and outputs to increase
clinical interpretability in one reading. Finally, we demonstrate the overall
superior performance of our model compared to prior arts on various chest X-ray
benchmarks across multi-tasks in both direct inference and finetuning settings.
Three radiologists further evaluate the generated reports against the recorded
ones, which also exhibit the enhanced explainability of our multi-task model.

基于多模态深度学习模型的临床应用中的多任务程序，我们提出了一个统一的变压器模型，通过整合定制化指令调整，将不同的视觉密集型任务统一在一个训练框架中，以增加临床解释性和在不同胸部 X 光基准测试中表现出的优越性能。

通过统一和定制指令调整，学习一种多任务变压器用于胸片解读

Learning A Multi-Task Transformer Via Unified And Customized Instruction  Tuning For Chest Radiograph Interpretation

We propose UniT, a Unified Transformer model to simultaneously learn the most
prominent tasks across different domains, ranging from object detection to
natural language understanding and multimodal reasoning. Based on the
transformer encoder-decoder architecture, our UniT model encodes each input
modality with an encoder and makes predictions on each task with a shared
decoder over the encoded input representations, followed by task-specific
output heads. The entire model is jointly trained end-to-end with losses from
each task. Compared to previous efforts on multi-task learning with
transformers, we share the same model parameters across all tasks instead of
separately fine-tuning task-specific models and handle a much higher variety of
tasks across different domains. In our experiments, we learn 7 tasks jointly
over 8 datasets, achieving strong performance on each task with significantly
fewer parameters. Our code is available in MMF at this https URL

我们提出了一种名为 UniT 的统一 Transformer 模型，该模型可以同时学习不同领域中最重要的任务，从目标检测到自然语言理解和多模态推理，在编码器 - 解码器架构的基础上，通过编码器对每个输入模态进行编码，并使用共享解码器对编码的输入表示进行每个任务的预测，然后是特定于任务的输出头。我们的实验证明，在 8 个数据集上联合学习 7 个任务，使用比以前的工作更少的参数，在每个任务上都获得了强大的性能。