This paper proposes a novel approach to learn commonsense from images,
instead of limited raw texts or costly constructed knowledge bases, for the
commonsense reasoning problem in NLP. Our motivation comes from the fact that
an image is worth a thousand words, where richer scene information could be
leveraged to help distill the commonsense knowledge, which is often hidden in
languages. Our approach, namely Loire, consists of two stages. In the first
stage, a bi-modal sequence-to-sequence approach is utilized to conduct the
scene layout generation task, based on a text representation model ViBERT. In
this way, the required visual scene knowledge, such as spatial relations, will
be encoded in ViBERT by the supervised learning process with some bi-modal data
like COCO. Then ViBERT is concatenated with a pre-trained language model to
perform the downstream commonsense reasoning tasks. Experimental results on two
commonsense reasoning problems, i.e. commonsense question answering and pronoun
resolution, demonstrate that Loire outperforms traditional language-based
methods. We also give some case studies to show what knowledge is learned from
images and explain how the generated scene layout helps the commonsense
reasoning process.

本文提出了一种从图像中学习常识的新方法，利用双模态序列到序列模型进行场景布局生成任务，并在通感问题回答和代词消解领域实验中证明了新方法的有效性。