Visual understanding goes well beyond object recognition. With one glance at
an image, we can effortlessly imagine the world beyond the pixels: for
instance, we can infer people's actions, goals, and mental states. While this
task is easy for humans, it is tremendously difficult for today's vision
systems, requiring higher-order cognition and commonsense reasoning about the
world. We formalize this task as Visual Commonsense Reasoning. Given a
challenging question about an image, a machine must answer correctly and then
provide a rationale justifying its answer.
Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA
problems derived from 110k movie scenes. The key recipe for generating
non-trivial and high-quality problems at scale is Adversarial Matching, a new
approach to transform rich annotations into multiple choice questions with
minimal bias. Experimental results show that while humans find VCR easy (over
90% accuracy), state-of-the-art vision models struggle (~45%).
To move towards cognition-level understanding, we present a new reasoning
engine, Recognition to Cognition Networks (R2C), that models the necessary
layered inferences for grounding, contextualization, and reasoning. R2C helps
narrow the gap between humans and machines (~65%); still, the challenge is far
from solved, and we provide analysis that suggests avenues for future work.

该研究论文旨在推进计算机视觉模型的视觉常识推理能力。研究者针对这一目标，提出一项包含 290k 多选题的新数据集 VCR，并使用基于对抗匹配的方法进行添加问题。通过引入新的推理引擎 - Recognition to Cognition 网络（R2C），该文章对计算机视觉模型的性能进行了分析，并提出未来研究的方向。