Large language models have recently shown human level performance on a
variety of reasoning tasks. However, the ability of these models to perform
complex visual reasoning has not been studied in detail yet. A key challenge in
many visual reasoning tasks is that the visual information needs to be tightly
integrated in the reasoning process. We propose to address this challenge by
drawing inspiration from human visual problem solving which depends on a
variety of low-level visual capabilities. It can often be cast as the three
step-process of ``Look, Remember, Reason'': visual information is incrementally
extracted using low-level visual routines in a step-by-step fashion until a
final answer is reached. We follow the same paradigm to enable existing large
language models, with minimal changes to the architecture, to solve visual
reasoning problems. To this end, we introduce rationales over the visual input
that allow us to integrate low-level visual capabilities, such as object
recognition and tracking, as surrogate tasks. We show competitive performance
on diverse visual reasoning tasks from the CLEVR, CATER, and ACRE datasets over
state-of-the-art models designed specifically for these tasks.

该研究旨在通过模仿人类视觉问题解决中的 “看、记住、推理” 模式，引入基于视觉输入的原理来整合低级视觉能力，使现有的大型语言模型能够在视觉推理问题上取得竞争性表现。