In this paper, we present our solution to a Multi-modal Algorithmic Reasoning
Task: SMART-101 Challenge. Different from the traditional visual
question-answering datasets, this challenge evaluates the abstraction,
deduction, and generalization abilities of neural networks in solving
visuolinguistic puzzles designed specifically for children in the 6-8 age
group. We employed a divide-and-conquer approach. At the data level, inspired
by the challenge paper, we categorized the whole questions into eight types and
utilized the llama-2-chat model to directly generate the type for each question
in a zero-shot manner. Additionally, we trained a yolov7 model on the icon45
dataset for object detection and combined it with the OCR method to recognize
and locate objects and text within the images. At the model level, we utilized
the BLIP-2 model and added eight adapters to the image encoder VIT-G to
adaptively extract visual features for different question types. We fed the
pre-constructed question templates as input and generated answers using the
flan-t5-xxl decoder. Under the puzzle splits configuration, we achieved an
accuracy score of 26.5 on the validation set and 24.30 on the private test set.

本文介绍了我们在 SMART-101 挑战中解决多模态算法推理任务的解决方案。我们使用分而治之的方法，在数据层面上将所有问题分为八类，并利用 llama-2-chat 模型以零射方式直接生成每个问题的类别。此外，我们在 icon45 数据集上训练了一个 yolov7 模型用于目标检测，并将其与 OCR 方法结合起来识别和定位图像中的对象和文本。在模型层面上，我们使用 BLIP-2 模型并为图像编码器 VIT-G 添加了八个适配器，以自适应地提取不同问题类型的视觉特征。我们将预先构建的问题模板作为输入，使用 flan-t5-xxl 解码器生成答案。在拼图分割配置下，我们在验证集上获得了 26.5 的准确率得分，并在私有测试集上获得了 24.30 的准确率得分。