The integration of thermal imaging data with Multimodal Large Language Models
(MLLMs) constitutes an exciting opportunity for improving the safety and
functionality of autonomous driving systems and many Intelligent Transportation
Systems (ITS) applications. This study investigates whether MLLMs can
understand complex images from RGB and thermal cameras and detect objects
directly. Our goals were to 1) assess the ability of the MLLM to learn from
information from various sets, 2) detect objects and identify elements in
thermal cameras, 3) determine whether two independent modality images show the
same scene, and 4) learn all objects using different modalities. The findings
showed that both GPT-4 and Gemini were effective in detecting and classifying
objects in thermal images. Similarly, the Mean Absolute Percentage Error (MAPE)
for pedestrian classification was 70.39% and 81.48%, respectively. Moreover,
the MAPE for bike, car, and motorcycle detection were 78.4%, 55.81%, and
96.15%, respectively. Gemini produced MAPE of 66.53%, 59.35% and 78.18%
respectively. This finding further demonstrates that MLLM can identify thermal
images and can be employed in advanced imaging automation technologies for ITS
applications.

研究表明，多模态大型语言模型（Multimodal Large Language Models，MLLMs）与热像数据的集成对提高自动驾驶系统和智能交通系统（ITS）应用的安全性和功能性构成了一个令人兴奋的机遇。该研究探讨了 MLLM 是否能够理解来自 RGB 和热像相机的复杂图像，并直接检测物体。结果表明，GPT-4 和 Gemini 在热像中检测和分类物体方面都非常有效，并且可以应用于 ITS 应用的先进成像自动化技术。