Multi-modal relation extraction (MMRE) is a challenging task that aims to
identify relations between entities in text leveraging image information.
Existing methods are limited by their neglect of the multiple entity pairs in
one sentence sharing very similar contextual information (ie, the same text and
image), resulting in increased difficulty in the MMRE task. To address this
limitation, we propose the Variational Multi-Modal Hypergraph Attention Network
(VM-HAN) for multi-modal relation extraction. Specifically, we first construct
a multi-modal hypergraph for each sentence with the corresponding image, to
establish different high-order intra-/inter-modal correlations for different
entity pairs in each sentence. We further design the Variational Hypergraph
Attention Networks (V-HAN) to obtain representational diversity among different
entity pairs using Gaussian distribution and learn a better hypergraph
structure via variational attention. VM-HAN achieves state-of-the-art
performance on the multi-modal relation extraction task, outperforming existing
methods in terms of accuracy and efficiency.

提出了一种用于多模态关系提取的变分多模态超图注意网络（VM-HAN），通过利用图像信息在文本中识别实体间的关系，构建了每个句子的多模态超图，并利用高斯分布实现了实体对之间的代表性多样性，从而在多模态关系提取任务中取得了最先进的性能。

多模态关系抽取的变分多模态超图注意力网络

Variational Multi-Modal Hypergraph Attention Network for Multi-Modal  Relation Extraction

Multi-Modal Relation Extraction (MMRE) aims at identifying the relation
between two entities in texts that contain visual clues. Rich visual content is
valuable for the MMRE task, but existing works cannot well model finer
associations among different modalities, failing to capture the truly helpful
visual information and thus limiting relation extraction performance. In this
paper, we propose a novel MMRE framework to better capture the deeper
correlations of text, entity pair, and image/objects, so as to mine more
helpful information for the task, termed as DGF-PT. We first propose a
prompt-based autoregressive encoder, which builds the associations of
intra-modal and inter-modal features related to the task, respectively by
entity-oriented and object-oriented prefixes. To better integrate helpful
visual information, we design a dual-gated fusion module to distinguish the
importance of image/objects and further enrich text representations. In
addition, a generative decoder is introduced with entity type restriction on
relations, better filtering out candidates. Extensive experiments conducted on
the benchmark dataset show that our approach achieves excellent performance
compared to strong competitors, even in the few-shot situation.

本文提出了一种新的多模态关系提取框架，称为 DGF-PT，该框架旨在更好地捕捉文本、实体对和图像 / 对象之间的更深层次的相关性，以挖掘更有用的信息，并且通过引入对实体类型的限制，更好地过滤候选关系。