Most existing visual reasoning tasks, such as CLEVR in VQA, ignore an
important factor, i.e.~transformation. They are solely defined to test how well
machines understand concepts and relations within static settings, like one
image. Such \textbf{state driven} visual reasoning has limitations in
reflecting the ability to infer the dynamics between different states, which
has shown to be equally important for human cognition in Piaget's theory. To
tackle this problem, we propose a novel \textbf{transformation driven} visual
reasoning (TVR) task. Given both the initial and final states, the target
becomes to infer the corresponding intermediate transformation. Following this
definition, a new synthetic dataset namely TRANCE is first constructed on the
basis of CLEVR, including three levels of settings, i.e.~Basic (single-step
transformation), Event (multi-step transformation), and View (multi-step
transformation with variant views). Next, we build another real dataset called
TRANCO based on COIN, to cover the loss of transformation diversity on TRANCE.
Inspired by human reasoning, we propose a three-staged reasoning framework
called TranNet, including observing, analyzing, and concluding, to test how
recent advanced techniques perform on TVR. Experimental results show that the
state-of-the-art visual reasoning models perform well on Basic, but are still
far from human-level intelligence on Event, View, and TRANCO. We believe the
proposed new paradigm will boost the development of machine visual reasoning.
More advanced methods and new problems need to be investigated in this
direction. The resource of TVR is available at
https://hongxin2019.github.io/TVR/.

本文提出了一种基于变换的视觉推理任务 (TVR)，并构建了 TRANCE 合成数据集以及基于 COIN 的 TRANCO 真实数据集，用于测试机器对于多步变换场景的推理能力。实验结果表明，目前先进的视觉推理模型在基本的单步变换上表现良好，但在多步变换，多视角下的场景以及 TRANCO 数据集上，机器的推理能力与人类差距较大。该研究的提出将有助于推动机器视觉推理的发展。