The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing attention within the research comminity. In this paper, we specifically focus on referring expression comprehension with commonsense knowledge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Transformers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitating identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art.

本研究提出了一种基于常识知识增强的Transformer框架(CK-Transformer)用于图像中多模态指称表述的理解，实验结果表明CK-Transformer在KB-Ref任务上实现了一项新的最优表现，相比现有技术提高了3.14%的准确性。

CK-Transformer：基于常识知识的Transformer模型用于指代表达理解