Pretrained vision-language models, such as CLIP, have demonstrated strong
generalization capabilities, making them promising tools in the realm of
zero-shot visual recognition. Visual relation detection (VRD) is a typical task
that identifies relationship (or interaction) types between object pairs within
an image. However, naively utilizing CLIP with prevalent class-based prompts
for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish
between different fine-grained relation types and it neglects essential spatial
information of two objects. To this end, we propose a novel method for
zero-shot VRD: RECODE, which solves RElation detection via COmposite
DEscription prompts. Specifically, RECODE first decomposes each predicate
category into subject, object, and spatial components. Then, it leverages large
language models (LLMs) to generate description-based prompts (or visual cues)
for each component. Different visual cues enhance the discriminability of
similar relation categories from different perspectives, which significantly
boosts performance in VRD. To dynamically fuse different cues, we further
introduce a chain-of-thought method that prompts LLMs to generate reasonable
weights for different visual cues. Extensive experiments on four VRD benchmarks
have demonstrated the effectiveness and interpretability of RECODE.

通过使用 RECODE 方法，利用预训练图像 - 语言模型，我们能够解决零样本可视关系检测中存在的问题，提高了关系检测的准确性和可解释性。

基于大型语言模型的复合视觉线索进行零样本视觉关系检测

Zero-shot Visual Relation Detection via Composite Visual Cues from Large  Language Models

Visual Relation Detection (VRD) aims to detect relationships between objects
for image understanding. Most existing VRD methods rely on thousands of
training samples of each relationship to achieve satisfactory performance. Some
recent papers tackle this problem by few-shot learning with elaborately
designed pipelines and pre-trained word vectors. However, the performance of
existing few-shot VRD models is severely hampered by the poor generalization
capability, as they struggle to handle the vast semantic diversity of visual
relationships. Nonetheless, humans have the ability to learn new relationships
with just few examples based on their knowledge. Inspired by this, we devise a
knowledge-augmented, few-shot VRD framework leveraging both textual knowledge
and visual relation knowledge to improve the generalization ability of few-shot
VRD. The textual knowledge and visual relation knowledge are acquired from a
pre-trained language model and an automatically constructed visual relation
knowledge graph, respectively. We extensively validate the effectiveness of our
framework. Experiments conducted on three benchmarks from the commonly used
Visual Genome dataset show that our performance surpasses existing
state-of-the-art models with a large improvement.

本研究提出了一个基于知识增强的少样本学习（few-shot learning）框架，利用预训练语言模型和自动构建的视觉关系知识图谱获取了文本知识和视觉关系知识，从而提高了少样本学习的实用性和性能。与现有最先进模型相比，在常用的 Visual Genome 数据集的三个基准测试中，本方法的性能大幅超越了现有最先进模型。