With massive explosion of social media such as Twitter and Instagram, people
daily share billions of multimedia posts, containing images and text.
Typically, text in these posts is short, informal and noisy, leading to
ambiguities which can be resolved using images. In this paper we explore
text-centric Named Entity Recognition task on these multimedia posts. We
propose an end to end model which learns a joint representation of a text and
an image. Our model extends multi-dimensional self attention technique, where
now image helps to enhance relationship between words. Experiments show that
our model is capable of capturing both textual and visual contexts with greater
accuracy, achieving state-of-the-art results on Twitter multimodal Named Entity
Recognition dataset.

本文在 Twitter 和 Instagram 等社交媒体大规模爆炸的情况下，探索了基于多媒体发布中关于实体识别的问题。我们提出了一种端到端模型，学习了文本和图像的联合表示。模型扩展了多维自我关注技术，其中现在图像有助于增强单词之间的关系。实验表明，我们的模型能够更精确地捕捉文本和视觉语境，实现了 Twitter 多模式命名实体识别数据集的最新结果。