Multimodal Entity Linking (MEL) is the task of mapping mentions with
multimodal contexts to the referent entities from a knowledge base (e.g.,
Wikipedia). Prior MEL methods mainly focus on designing complex multimodal
interaction mechanisms and require fine-tuning all model parameters, which can
be prohibitively costly and difficult to scale in the era of Large Language
Models (LLMs). In this work, we propose GEMEL, a simple yet effective
Generative Multimodal Entity Linking method, which leverages the capabilities
of LLMs from large-scale pre-training to directly generate target entity names.
We keep the vision and language model frozen and only train a linear layer to
enable cross-modality interactions. To adapt LLMs to the MEL task, we take
advantage of the emerging in-context learning (ICL) capability of LLMs by
retrieving multimodal instances as demonstrations. Extensive experiments show
that with only ~0.3% of the model parameters fine-tuned, GEMEL achieves
state-of-the-art results on two well-established MEL datasets (4.1% accuracy
gains on WikiDiverse and 15.4% accuracy gains on WikiMEL). Our approach is
compatible with any off-the-shelf language model, paving the way towards an
efficient and general solution for utilizing LLMs in the MEL task.

在大规模预训练模型的基础上，提出了一种基于自然语言生成的简单而有效的跨模态实体链接方法，通过使用 in-context learning 能力以及取回多模态实例作为演示来自适应语言模型来解决昂贵且难以扩展的困境。