Multi-modal named entity recognition (NER) and relation extraction (RE) aim
to leverage relevant image information to improve the performance of NER and
RE. Most existing efforts largely focused on directly extracting potentially
useful information from images (such as pixel-level features, identified
objects, and associated captions). However, such extraction processes may not
be knowledge aware, resulting in information that may not be highly relevant.
In this paper, we propose a novel Multi-modal Retrieval based framework (MoRe).
MoRe contains a text retrieval module and an image-based retrieval module,
which retrieve related knowledge of the input text and image in the knowledge
corpus respectively. Next, the retrieval results are sent to the textual and
visual models respectively for predictions. Finally, a Mixture of Experts (MoE)
module combines the predictions from the two models to make the final decision.
Our experiments show that both our textual model and visual model can achieve
state-of-the-art performance on four multi-modal NER datasets and one
multi-modal RE dataset. With MoE, the model performance can be further improved
and our analysis demonstrates the benefits of integrating both textual and
visual cues for such tasks.

本文提出了 MoRe 框架，使用多模态检索方法，结合文本和图像信息进行命名实体识别和关系抽取，并使用 Mixture of Experts 模块以获得最终决策。在实验中，MoRe 模型达到了四个多模态实体识别数据集和一个多模态关系抽取数据集的最佳性能，证明了结合文本和视觉提示在这些任务中对提高性能具有实际意义。