Having revolutionized natural language processing (NLP) applications, large
language models (LLMs) are expanding into the realm of multimodal inputs. Owing
to their ability to interpret images, multimodal LLMs (MLLMs) have been
primarily used for vision-language tasks. Currently, MLLMs have not yet been
extended for domain-specific visual tasks, which require a more explicit
understanding of visual information. We developed a method to transform
domain-specific visual and vision-language datasets into a unified question
answering format called Visual Question Answering Instruction (VQA-IN), thereby
extending MLLM to domain-specific tasks. The VQA-IN was applied to train
multiple MLLM architectures using smaller versions of LLMs (sLLMs). The
experimental results indicated that the proposed method achieved a high score
metric on domainspecific visual tasks while also maintaining its performance on
vision-language tasks in a multitask manner.

使用大型语言模型和多模态语言模型，我们开发了一种方法将特定领域的视觉和视觉 - 语言数据集转化为统一的问答格式，从而扩展了多模态语言模型用于特定领域任务，实验结果表明该方法在特定领域的视觉任务和视觉 - 语言任务上达到了高分数指标并保持了多任务的性能。