The task of Visual Relationship Recognition (VRR) aims to identify
relationships between two interacting objects in an image and is particularly
challenging due to the widely-spread and highly imbalanced distribution of
<subject, relation, object> triplets. To overcome the resultant performance
bias in existing VRR approaches, we introduce DiffAugment -- a method which
first augments the tail classes in the linguistic space by making use of
WordNet and then utilizes the generative prowess of Diffusion Models to expand
the visual space for minority classes. We propose a novel hardness-aware
component in diffusion which is based upon the hardness of each <S,R,O> triplet
and demonstrate the effectiveness of hardness-aware diffusion in generating
visual embeddings for the tail classes. We also propose a novel subject and
object based seeding strategy for diffusion sampling which improves the
discriminative capability of the generated visual embeddings. Extensive
experimentation on the GQA-LT dataset shows favorable gains in the
subject/object and relation average per-class accuracy using Diffusion
augmented samples.

Visual Relationship Recognition (VRR) using DiffAugment and Diffusion Models to address the imbalanced distribution of triplets, introducing a hardness-aware component and a subject/object-based seeding strategy, improving per-class accuracy on the GQA-LT dataset.