Entities can be expressed in diverse formats, such as texts, images, or
column names and cell values in tables. While existing entity linking (EL)
models work well on per modality configuration, such as text-only EL, visual
grounding, or schema linking, it is more challenging to design a unified model
for diverse modality configurations. To bring various modality configurations
together, we constructed a benchmark for diverse-modal EL (DMEL) from existing
EL datasets, covering all three modalities including text, image, and table. To
approach the DMEL task, we proposed a generative diverse-modal model (GDMM)
following a multimodal-encoder-decoder paradigm. Pre-training \Model with rich
corpora builds a solid foundation for DMEL without storing the entire KB for
inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming
state-of-the-art task-specific EL models by 8.51 F1 score on average.
Additionally, extensive error analyses are conducted to highlight the
challenges of DMEL, facilitating future research on this task.

提出了一种基于多模态编码器 - 解码器范式的生成模型，用于对广泛多种格式的实体进行链接，并在现有 EL 数据集中建立了覆盖文本、图像和表格三种多模态的基准，经预训练和微调后能表现比特定任务 EL 模型更强的 DMEL 基线，相比其它方法平均提高 8.51 的 F1 得分。