Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of
attention. Most of the work utilizes image information through region-level
visual representations obtained from a pretrained object detector and relies on
an attention mechanism to model the interactions between image and text
representations. However, it is difficult to model such interactions as image
and text representations are trained separately on the data of their respective
modality and are not aligned in the same space. As text representations take
the most important role in MNER, in this paper, we propose {\bf I}mage-{\bf
t}ext {\bf A}lignments (ITA) to align image features into the textual space, so
that the attention mechanism in transformer-based pretrained textual embeddings
can be better utilized. ITA first aligns the image into regional object tags,
image-level captions and optical characters as visual contexts, concatenates
them with the input texts as a new cross-modal input, and then feeds it into a
pretrained textual embedding model. This makes it easier for the attention
module of a pretrained textual embedding model to model the interaction between
the two modalities since they are both represented in the textual space. ITA
further aligns the output distributions predicted from the cross-modal input
and textual input views so that the MNER model can be more practical in dealing
with text-only inputs and robust to noises from images. In our experiments, we
show that ITA models can achieve state-of-the-art accuracy on multi-modal Named
Entity Recognition datasets, even without image information.

本文提出了一种基于图像与文本对齐的多模态命名实体识别技术，通过将图像特征与文本信息在文本空间中对齐，将两者之间的交互作用结合在一起，以此提高命名实体识别的准确性。