Multimodal entity linking (MEL) task, which aims at resolving ambiguous
mentions to a multimodal knowledge graph, has attracted wide attention in
recent years. Though large efforts have been made to explore the complementary
effect among multiple modalities, however, they may fail to fully absorb the
comprehensive expression of abbreviated textual context and implicit visual
indication. Even worse, the inevitable noisy data may cause inconsistency of
different modalities during the learning process, which severely degenerates
the performance. To address the above issues, in this paper, we propose a novel
Multi-GraIned Multimodal InteraCtion Network $\textbf{(MIMIC)}$ framework for
solving the MEL task. Specifically, the unified inputs of mentions and entities
are first encoded by textual/visual encoders separately, to extract global
descriptive features and local detailed features. Then, to derive the
similarity matching score for each mention-entity pair, we device three
interaction units to comprehensively explore the intra-modal interaction and
inter-modal fusion among features of entities and mentions. In particular,
three modules, namely the Text-based Global-Local interaction Unit (TGLU),
Vision-based DuaL interaction Unit (VDLU) and Cross-Modal Fusion-based
interaction Unit (CMFU) are designed to capture and integrate the fine-grained
representation lying in abbreviated text and implicit visual cues. Afterwards,
we introduce a unit-consistency objective function via contrastive learning to
avoid inconsistency and model degradation. Experimental results on three public
benchmark datasets demonstrate that our solution outperforms various
state-of-the-art baselines, and ablation studies verify the effectiveness of
designed modules.

本文提出了一种名为多维多模态交互网络（MIMIC）的框架，用于解决多模态实体链接（MEL）任务，并通过对相互作用单元和对比学习的设计，将简洁文本和隐含视觉线索的精细表示结合起来，实现了更好的性能，超过了各种最先进的基准模型。