Large-scale multi-modal pre-training models such as CLIP and PaLI exhibit
strong generalization on various visual domains and tasks. However, existing
image classification benchmarks often evaluate recognition on a specific domain
(e.g., outdoor images) or a specific task (e.g., classifying plant species),
which falls short of evaluating whether pre-trained foundational models are
universal visual recognizers. To address this, we formally present the task of
Open-domain Visual Entity recognitioN (OVEN), where a model need to link an
image onto a Wikipedia entity with respect to a text query. We construct
OVEN-Wiki by re-purposing 14 existing datasets with all labels grounded onto
one single label space: Wikipedia entities. OVEN challenges models to select
among six million possible Wikipedia entities, making it a general visual
recognition benchmark with the largest number of labels. Our study on
state-of-the-art pre-trained models reveals large headroom in generalizing to
the massive-scale label space. We show that a PaLI-based auto-regressive visual
recognition model performs surprisingly well, even on Wikipedia entities that
have never been seen during fine-tuning. We also find existing pretrained
models yield different strengths: while PaLI-based models obtain higher overall
performance, CLIP-based models are better at recognizing tail entities.

本文正式提出了开放域视觉实体识别（OVEN）这一任务，为评估预训练模型的普适性，构建了一个最大标签数量的通用视觉识别基准数据集。该数据集将 14 个现有的数据集标签全部调整为维基百科实体，其中包含了六百万个可能的实体，我们在此基础上测试了一系列领先的预训练模型，并发现不同的模型各有千秋。