In this paper, the solution of HYU MLLAB KT Team to the Multimodal
Algorithmic Reasoning Task: SMART-101 CVPR 2024 Challenge is presented. Beyond
conventional visual question-answering problems, the SMART-101 challenge aims
to achieve human-level multimodal understanding by tackling complex
visio-linguistic puzzles designed for children in the 6-8 age group. To solve
this problem, we suggest two main ideas. First, to utilize the reasoning
ability of a large-scale language model (LLM), the given visual cues (images)
are grounded in the text modality. For this purpose, we generate highly
detailed text captions that describe the context of the image and use these
captions as input for the LLM. Second, due to the nature of puzzle images,
which often contain various geometric visual patterns, we utilize an object
detection algorithm to ensure these patterns are not overlooked in the
captioning process. We employed the SAM algorithm, which can detect
various-size objects, to capture the visual features of these geometric
patterns and used this information as input for the LLM. Under the puzzle split
configuration, we achieved an option selection accuracy Oacc of 29.5 on the
test set and a weighted option selection accuracy (WOSA) of 27.1 on the
challenge set.

在本论文中，介绍了 HYU MLLAB KT 团队解决 SMART-101 CVPR 2024 Challenge：多模态算法推理任务。为了解决这个问题，我们提出了两个主要思路：利用大规模语言模型 (LLM) 的推理能力，将给定的视觉线索 (图像) 落实到文本模态中，并生成描述图像上下文的详细文本描述，然后将这些描述作为 LLM 的输入；为了确保不会忽视包含各种几何视觉模式的拼图图像的特性，我们利用目标检测算法来捕捉这些几何模式的视觉特征，并将这些信息作为 LLM 的输入。在拼图分割配置下，在测试集上实现了 29.5 的选项选择准确率 (Oacc)，在挑战集上实现了 27.1 的加权选项选择准确率 (WOSA)。