Instruction finetuning is a popular paradigm to align large language models
(LLM) with human intent. Despite its popularity, this idea is less explored in
improving the LLMs to align existing foundation models with scientific
disciplines, concepts and goals. In this work, we present SciTune as a tuning
framework to improve the ability of LLMs to follow scientific multimodal
instructions. To test our methodology, we use a human-generated scientific
instruction tuning dataset and train a large multimodal model LLaMA-SciTune
that connects a vision encoder and LLM for science-focused visual and language
understanding. In comparison to the models that are finetuned with machine
generated data only, LLaMA-SciTune surpasses human performance on average and
in many sub-categories on the ScienceQA benchmark.

本研究提出了 SciTune 框架，通过调整 Large Language Models，与科学学科，概念和目标相符合，提高 LLMs 的理解科学多模态指令的能力，并通过 ScienceQA 基准测试表明，与仅使用机器生成数据调整的模型相比，LLaMA-SciTune 在各个子类别上均优于人类表现。