Large-scale multi-modal pre-training models such as CLIP and PaLI exhibit
strong generalization on various visual domains and tasks. However, existing
image classification benchmarks often evaluate recognition on a specific domain
(e.g., outdoor images) or a specific task (e.g., classifying plant species),
which falls short of evaluating whether pre-trained foundational models are
universal visual recognizers. To address this, we formally present the task of
Open-domain Visual Entity recognitioN (OVEN), where a model need to link an
image onto a Wikipedia entity with respect to a text query. We construct
OVEN-Wiki by re-purposing 14 existing datasets with all labels grounded onto
one single label space: Wikipedia entities. OVEN challenges models to select
among six million possible Wikipedia entities, making it a general visual
recognition benchmark with the largest number of labels. Our study on
state-of-the-art pre-trained models reveals large headroom in generalizing to
the massive-scale label space. We show that a PaLI-based auto-regressive visual
recognition model performs surprisingly well, even on Wikipedia entities that
have never been seen during fine-tuning. We also find existing pretrained
models yield different strengths: while PaLI-based models obtain higher overall
performance, CLIP-based models are better at recognizing tail entities.

本文正式提出了开放域视觉实体识别（OVEN）这一任务，为评估预训练模型的普适性，构建了一个最大标签数量的通用视觉识别基准数据集。该数据集将 14 个现有的数据集标签全部调整为维基百科实体，其中包含了六百万个可能的实体，我们在此基础上测试了一系列领先的预训练模型，并发现不同的模型各有千秋。

开放领域视觉实体识别：朝着识别数百万维基百科实体迈进

Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities

Recent studies have shown that multilingual pretrained language models can be
effectively improved with cross-lingual alignment information from Wikipedia
entities. However, existing methods only exploit entity information in
pretraining and do not explicitly use entities in downstream tasks. In this
study, we explore the effectiveness of leveraging entity representations for
downstream cross-lingual tasks. We train a multilingual language model with 24
languages with entity representations and show the model consistently
outperforms word-based pretrained models in various cross-lingual transfer
tasks. We also analyze the model and the key insight is that incorporating
entity representations into the input allows us to extract more
language-agnostic features. We also evaluate the model with a multilingual
cloze prompt task with the mLAMA dataset. We show that entity-based prompt
elicits correct factual knowledge more likely than using only word
representations. Our source code and pretrained models are available at
this https URL

此篇论文研究了如何利用来自 Wikipedia 实体的跨语言对齐信息来提高多语言预训练语言模型的效果，并探索了在下游跨语言任务中使用实体表示方法的有效性。

mLUKE: 多语言预训练语言模型中实体表示的强大力量

mLUKE: The Power of Entity Representations in Multilingual Pretrained  Language Models

We study the problem of entity salience by proposing the design and
implementation of SWAT, a system that identifies the salient Wikipedia entities
occurring in an input document. SWAT consists of several modules that are able
to detect and classify on-the-fly Wikipedia entities as salient or not, based
on a large number of syntactic, semantic and latent features properly extracted
via a supervised process which has been trained over millions of examples drawn
from the New York Times corpus. The validation process is performed through a
large experimental assessment, eventually showing that SWAT improves known
solutions over all publicly available datasets. We release SWAT via an API that
we describe and comment in the paper in order to ease its use in other
software.

本文介绍了一个称为 SWAT 的系统，该系统通过监督过程检测和分类维基百科中的实体，最终使用 API 的方式公开。经过大量实验对比，证明 SWAT 在提高实体关注度方面比目前已有的公开数据集更为有效。