Entity-aware image captioning aims to describe named entities and events
related to the image by utilizing the background knowledge in the associated
article. This task remains challenging as it is difficult to learn the
association between named entities and visual cues due to the long-tail
distribution of named entities. Furthermore, the complexity of the article
brings difficulty in extracting fine-grained relationships between entities to
generate informative event descriptions about the image. To tackle these
challenges, we propose a novel approach that constructs a multi-modal knowledge
graph to associate the visual objects with named entities and capture the
relationship between entities simultaneously with the help of external
knowledge collected from the web. Specifically, we build a text sub-graph by
extracting named entities and their relationships from the article, and build
an image sub-graph by detecting the objects in the image. To connect these two
sub-graphs, we propose a cross-modal entity matching module trained using a
knowledge base that contains Wikipedia entries and the corresponding images.
Finally, the multi-modal knowledge graph is integrated into the captioning
model via a graph attention mechanism. Extensive experiments on both GoodNews
and NYTimes800k datasets demonstrate the effectiveness of our method.

该研究提出一种构建多模态知识图谱，将视觉对象和命名实体联系起来并同时捕捉实体之间的关系在内的新方法，以帮助名实体感知图像并生成更具信息量的事件描述。