Mining structured knowledge from tweets using named entity recognition (NER)
can be beneficial for many downstream applications such as recommendation and
intention under standing. With tweet posts tending to be multimodal, multimodal
named entity recognition (MNER) has attracted more attention. In this paper, we
propose a novel approach, which can dynamically align the image and text
sequence and achieve the multi-level cross-modal learning to augment textual
word representation for MNER improvement. To be specific, our framework can be
split into three main stages: the first stage focuses on intra-modality
representation learning to derive the implicit global and local knowledge of
each modality, the second evaluates the relevance between the text and its
accompanying image and integrates different grained visual information based on
the relevance, the third enforces semantic refinement via iterative cross-modal
interactions and co-attention. We conduct experiments on two open datasets, and
the results and detailed analysis demonstrate the advantage of our model.

本文针对 Tweet 这类多模态数据，提出一种能够动态对齐图像和文本顺序，实现多级跨模态学习，以提高命名实体识别（MNER）的文本表示的新方法。实验结果及详细分析表明了模型的优势。