In this paper, we explore the potential of Vision-Language Models (VLMs),
specifically CLIP, in predicting visual object relationships, which involves
interpreting visual features from images into language-based relations. Current
state-of-the-art methods use complex graphical models that utilize language
cues and visual features to address this challenge. We hypothesize that the
strong language priors in CLIP embeddings can simplify these graphical models
paving for a simpler approach. We adopt the UVTransE relation prediction
framework, which learns the relation as a translational embedding with subject,
object, and union box embeddings from a scene. We systematically explore the
design of CLIP-based subject, object, and union-box representations within the
UVTransE framework and propose CREPE (CLIP Representation Enhanced Predicate
Estimation). CREPE utilizes text-based representations for all three bounding
boxes and introduces a novel contrastive training strategy to automatically
infer the text prompt for union-box. Our approach achieves state-of-the-art
performance in predicate estimation, mR@5 27.79, and mR@20 31.95 on the Visual
Genome benchmark, achieving a 15.3\% gain in performance over recent
state-of-the-art at mR@20. This work demonstrates CLIP's effectiveness in
object relation prediction and encourages further research on VLMs in this
challenging domain.

本文介绍了基于 CLIP 的模型在预测物体关系方面的潜力及对 UVTransE 框架进行了优化，提出了 CLIP Representation Enhanced Predicate Estimation (CREPE) 方法，通过对三个边界框的文本表示进行对比训练，取得了 state-of-the-art 的预测成果。