Multi-Modal Knowledge Graphs (MMKGs) have proven valuable for various
downstream tasks. However, scaling them up is challenging because building
large-scale MMKGs often introduces mismatched images (i.e., noise). Most
entities in KGs belong to the long tail, meaning there are few images of them
available online. This scarcity makes it difficult to determine whether a found
image matches the entity. To address this, we draw on the Triangle of Reference
Theory and suggest enhancing vision-language models with concept guidance.
Specifically, we introduce COG, a two-stage framework with COncept-Guided
vision-language models. The framework comprises a Concept Integration module,
which effectively identifies image-text pairs of long-tailed entities, and an
Evidence Fusion module, which offers explainability and enables human
verification. To demonstrate the effectiveness of COG, we create a dataset of
25k image-text pairs of long-tailed entities. Our comprehensive experiments
show that COG not only improves the accuracy of recognizing long-tailed
image-text pairs compared to baselines but also offers flexibility and
explainability.

Multi-Modal Knowledge Graphs (MMKGs) have proven valuable for various downstream tasks. To address the challenge of building large-scale MMKGs with mismatched images, this paper introduces COG, a framework that enhances vision-language models with concept guidance, effectively identifying image-text pairs of long-tailed entities and offering flexibility and explainability.

照亮阴影：用概念引导的视觉语言模型增强长尾实体引地

Light Up the Shadows: Enhance Long-Tailed Entity Grounding with  Concept-Guided Vision-Language Models

Although biomedical entity linking (BioEL) has made significant progress with
pre-trained language models, challenges still exist for fine-grained and
long-tailed entities. To address these challenges, we present BioELQA, a novel
model that treats Biomedical Entity Linking as Multiple Choice Question
Answering. BioELQA first obtains candidate entities with a fast retriever,
jointly presents the mention and candidate entities to a generator, and then
outputs the predicted symbol associated with its chosen entity. This
formulation enables explicit comparison of different candidate entities, thus
capturing fine-grained interactions between mentions and entities, as well as
among entities themselves. To improve generalization for long-tailed entities,
we retrieve similar labeled training instances as clues and concatenate the
input with retrieved instances for the generator. Extensive experimental
results show that BioELQA outperforms state-of-the-art baselines on several
datasets.

Biomedical entity linking is improved using BioELQA, a model that treats it as multiple choice question answering, capturing fine-grained interactions and addressing challenges with long-tailed entities.