Despite the recent advancements in Multi-modal Large Language Models (MLLMs),
understanding inter-object relations, i.e., interactions or associations
between distinct objects, remains a major challenge for such models. This issue
significantly hinders their advanced reasoning capabilities and is primarily
due to the lack of large-scale, high-quality, and diverse multi-modal data
essential for training and evaluating MLLMs. In this paper, we provide a
taxonomy of inter-object relations and introduce Multi-Modal Relation
Understanding (MMRel), a comprehensive dataset designed to bridge this gap by
providing large-scale, high-quality and diverse data for studying inter-object
relations with MLLMs. MMRel features three distinctive attributes: (i) It
includes over 15K question-answer pairs, which are sourced from three distinct
domains, ensuring large scale and high diversity; (ii) It contains a subset
featuring highly unusual relations, on which MLLMs often fail due to
hallucinations, thus are very challenging; (iii) It provides manually verified
high-quality labels for inter-object relations. Thanks to these features, MMRel
is ideal for evaluating MLLMs on relation understanding, as well as being used
to fine-tune MLLMs to enhance relation understanding and even benefit overall
performance in various vision-language tasks. Extensive experiments on various
popular MLLMs validate the effectiveness of MMRel. Both MMRel dataset and the
complete labeling scripts have been made publicly available.

本研究提出了多模态关系理解（MMRel）数据集，其中包含大规模、高质量和多样化的数据，用于研究多模态大型语言模型在关系理解方面的能力，并证明了 MMRel 数据集的有效性。

MMRel：MLLM 时代的关系理解数据集和基准

MMRel: A Relation Understanding Dataset and Benchmark in the MLLM Era

Weakly supervised grounded image captioning (WSGIC) aims to generate the
caption and ground (localize) predicted object words in the input image without
using bounding box supervision. Recent two-stage solutions mostly apply a
bottom-up pipeline: (1) first apply an off-the-shelf object detector to encode
the input image into multiple region features; (2) and then leverage a
soft-attention mechanism for captioning and grounding. However, object
detectors are mainly designed to extract object semantics (i.e., the object
category). Besides, they break down the structural images into pieces of
individual proposals. As a result, the subsequent grounded captioner is often
overfitted to find the correct object words, while overlooking the relation
between objects (e.g., what is the person doing?), and selecting incompatible
proposal regions for grounding. To address these difficulties, we propose a
one-stage weakly supervised grounded captioner that directly takes the RGB
image as input to perform captioning and grounding at the top-down image level.
In addition, we explicitly inject a relation module into our one-stage
framework to encourage the relation understanding through multi-label
classification. The relation semantics aid the prediction of relation words in
the caption. We observe that the relation words not only assist the grounded
captioner in generating a more accurate caption but also improve the grounding
performance. We validate the effectiveness of our proposed method on two
challenging datasets (Flick30k Entities captioning and MSCOCO captioning). The
experimental results demonstrate that our method achieves state-of-the-art
grounding performance.

本研究提出了一种一阶段的弱监督图像标题生成方法，通过注入关系模块来鼓励多标签分类中的关系理解，从而提高了关注性能和字幕生成准确性，并在两个具有挑战性的数据集上验证了该方法的有效性。

自上而下视角的基于图像实体的图像字幕生成

Grounded Image Captioning in Top-down View

Contrastively trained vision-language models have achieved remarkable
progress in vision and language representation learning, leading to
state-of-the-art models for various downstream multimodal tasks. However,
recent research has highlighted severe limitations of these models in their
ability to perform compositional reasoning over objects, attributes, and
relations. Scene graphs have emerged as an effective way to understand images
compositionally. These are graph-structured semantic representations of images
that contain objects, their attributes, and relations with other objects in a
scene. In this work, we consider the scene graph parsed from text as a proxy
for the image scene graph and propose a graph decomposition and augmentation
framework along with a coarse-to-fine contrastive learning objective between
images and text that aligns sentences of various complexities to the same
image. Along with this, we propose novel negative mining techniques in the
scene graph space for improving attribute binding and relation understanding.
Through extensive experiments, we demonstrate the effectiveness of our approach
that significantly improves attribute binding, relation understanding,
systematic generalization, and productivity on multiple recently proposed
benchmarks (For example, improvements upto $18\%$ for systematic
generalization, $16.5\%$ for relation understanding over a strong baseline),
while achieving similar or better performance than CLIP on various general
multimodal tasks.

通过构建图像场景的文本表示形式，提出了一种图形分解和增强框架以进行对比学习，同时提出了一种用于改善 scene graph 空间中属性绑定和关系理解的负样本挖掘技术，通过大量实验证明了该方法的有效性。