Even from a single frame of a still image, people can reason about the
dynamic story of the image before, after, and beyond the frame. For example,
given an image of a man struggling to stay afloat in water, we can reason that
the man fell into the water sometime in the past, the intent of that man at the
moment is to stay alive, and he will need help in the near future or else he
will get washed away. We propose VisualComet, the novel framework of visual
commonsense reasoning tasks to predict events that might have happened before,
events that might happen next, and the intents of the people at present. To
support research toward visual commonsense reasoning, we introduce the first
large-scale repository of Visual Commonsense Graphs that consists of over 1.4
million textual descriptions of visual commonsense inferences carefully
annotated over a diverse set of 60,000 images, each paired with short video
summaries of before and after. In addition, we provide person-grounding (i.e.,
co-reference links) between people appearing in the image and people mentioned
in the textual commonsense descriptions, allowing for tighter integration
between images and text. We establish strong baseline performances on this task
and demonstrate that integration between visual and textual commonsense
reasoning is the key and wins over non-integrative alternatives.

提出了一种名为 VisualComet 的框架，用于预测图片中可能发生的事件、人物意图和接下来可能发生的事件，并建立了一个包含超过 1.4 百万个文本描述和各种图像的广泛集合。此外，研究者在图片和文本之间建立了紧密的链接，以确保在视觉和文本上的共同推理取得了强劲的基础表现，证明了集成是关键。