While Large Language Models (LLMs) can achieve human-level performance in
various tasks, they continue to face challenges when it comes to effectively
tackling multi-step physics reasoning tasks. To identify the shortcomings of
existing models and facilitate further research in this area, we curated a
novel dataset, MM-PhyQA, which comprises well-constructed, high schoollevel
multimodal physics problems. By evaluating the performance of contemporary LLMs
that are publicly available, both with and without the incorporation of
multimodal elements in these problems, we aim to shed light on their
capabilities. For generating answers for questions consisting of multimodal
input (in this case, images and text) we employed Zero-shot prediction using
GPT-4 and utilized LLaVA (LLaVA and LLaVA-1.5), the latter of which were
fine-tuned on our dataset. For evaluating the performance of LLMs consisting
solely of textual input, we tested the performance of the base and fine-tuned
versions of the Mistral-7B and LLaMA2-7b models. We also showcased the
performance of the novel Multi-Image Chain-of-Thought (MI-CoT) Prompting
technique, which when used to train LLaVA-1.5 13b yielded the best results when
tested on our dataset, with superior scores in most metrics and the highest
accuracy of 71.65% on the test set.

利用现有模型对多步骤物理推理任务进行评估，通过引入多模态元素的高中级别物理问题的新数据集 MM-PhyQA，测试了包括 GPT-4 和 LLaVA 在内的多个大型语言模型的性能，并展示了基于 MI-CoT 提示技术的 LLaVA-1.5 13b 模型在测试集上的最佳结果，具有最高的 71.65% 准确率。