In this paper, the solution of HYU MLLAB KT Team to the Multimodal Algorithmic Reasoning Task: SMART-101 CVPR 2024 Challenge is presented. Beyond conventional visual question-answering problems, the SMART-101 challenge aims to achieve human-level multimodal understanding by tackling complex visio-linguistic puzzles designed for children in the 6-8 age group. To solve this problem, we suggest two main ideas. First, to utilize the reasoning ability of a large-scale language model (LLM), the given visual cues (images) are grounded in the text modality. For this purpose, we generate highly detailed text captions that describe the context of the image and use these captions as input for the LLM. Second, due to the nature of puzzle images, which often contain various geometric visual patterns, we utilize an object detection algorithm to ensure these patterns are not overlooked in the captioning process. We employed the SAM algorithm, which can detect various-size objects, to capture the visual features of these geometric patterns and used this information as input for the LLM. Under the puzzle split configuration, we achieved an option selection accuracy Oacc of 29.5 on the test set and a weighted option selection accuracy (WOSA) of 27.1 on the challenge set.

在本论文中，介绍了HYU MLLAB KT团队解决SMART-101 CVPR 2024 Challenge：多模态算法推理任务。为了解决这个问题，我们提出了两个主要思路：利用大规模语言模型(LLM)的推理能力，将给定的视觉线索(图像)落实到文本模态中，并生成描述图像上下文的详细文本描述，然后将这些描述作为LLM的输入；为了确保不会忽视包含各种几何视觉模式的拼图图像的特性，我们利用目标检测算法来捕捉这些几何模式的视觉特征，并将这些信息作为LLM的输入。在拼图分割配置下，在测试集上实现了29.5的选项选择准确率(Oacc)，在挑战集上实现了27.1的加权选项选择准确率(WOSA)。

CVPR2024多模态算法推理任务SMART-101挑战解决方案