Recent times have witnessed an increasing number of applications of deep
neural networks towards solving tasks that require superior cognitive
abilities, e.g., playing Go, generating art, question answering (such as
ChatGPT), etc. Such a dramatic progress raises the question: how generalizable
are neural networks in solving problems that demand broad skills? To answer
this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task
and the associated SMART-101 dataset, for evaluating the abstraction,
deduction, and generalization abilities of neural networks in solving
visuo-linguistic puzzles designed specifically for children in the 6-8 age
group. Our dataset consists of 101 unique puzzles; each puzzle comprises a
picture and a question, and their solution needs a mix of several elementary
skills, including arithmetic, algebra, and spatial reasoning, among others. To
scale our dataset towards training deep neural networks, we programmatically
generate entirely new instances for each puzzle while retaining their solution
algorithm. To benchmark the performance on the SMART-101 dataset, we propose a
vision and language meta-learning model using varied state-of-the-art backbone
neural networks. Our experiments reveal that while powerful deep models offer
reasonable performances on puzzles that they are trained on, they are not
better than random accuracy when analyzed for generalization. We also evaluate
the recent ChatGPT large language model on a subset of our dataset and find
that while ChatGPT produces convincing reasoning abilities, the answers are
often incorrect.

本文旨在研究神经网络的广泛性，提出 SMART 算法推理任务和关联的 SMART-101 数据集以评估神经网络在 6-8 岁儿童视觉语言谜题领域中的抽象、推理和概括能力，结果表明，神经网络在特定领域的表现合理，但不具有良好的广泛性，ChatGPT 虽然具有令人信服的推理能力，但答案常常是错误的。