Multi-modal relation extraction (MMRE) is a challenging task that aims to
identify relations between entities in text leveraging image information.
Existing methods are limited by their neglect of the multiple entity pairs in
one sentence sharing very similar contextual information (ie, the same text and
image), resulting in increased difficulty in the MMRE task. To address this
limitation, we propose the Variational Multi-Modal Hypergraph Attention Network
(VM-HAN) for multi-modal relation extraction. Specifically, we first construct
a multi-modal hypergraph for each sentence with the corresponding image, to
establish different high-order intra-/inter-modal correlations for different
entity pairs in each sentence. We further design the Variational Hypergraph
Attention Networks (V-HAN) to obtain representational diversity among different
entity pairs using Gaussian distribution and learn a better hypergraph
structure via variational attention. VM-HAN achieves state-of-the-art
performance on the multi-modal relation extraction task, outperforming existing
methods in terms of accuracy and efficiency.

提出了一种用于多模态关系提取的变分多模态超图注意网络（VM-HAN），通过利用图像信息在文本中识别实体间的关系，构建了每个句子的多模态超图，并利用高斯分布实现了实体对之间的代表性多样性，从而在多模态关系提取任务中取得了最先进的性能。

多模态关系抽取的变分多模态超图注意力网络

Variational Multi-Modal Hypergraph Attention Network for Multi-Modal  Relation Extraction

Text-video retrieval is a challenging task that aims to identify relevant
videos given textual queries. Compared to conventional textual retrieval, the
main obstacle for text-video retrieval is the semantic gap between the textual
nature of queries and the visual richness of video content. Previous works
primarily focus on aligning the query and the video by finely aggregating
word-frame matching signals. Inspired by the human cognitive process of
modularly judging the relevance between text and video, the judgment needs
high-order matching signal due to the consecutive and complex nature of video
contents. In this paper, we propose chunk-level text-video matching, where the
query chunks are extracted to describe a specific retrieval unit, and the video
chunks are segmented into distinct clips from videos. We formulate the
chunk-level matching as n-ary correlations modeling between words of the query
and frames of the video and introduce a multi-modal hypergraph for n-ary
correlation modeling. By representing textual units and video frames as nodes
and using hyperedges to depict their relationships, a multi-modal hypergraph is
constructed. In this way, the query and the video can be aligned in a
high-order semantic space. In addition, to enhance the model's generalization
ability, the extracted features are fed into a variational inference component
for computation, obtaining the variational representation under the Gaussian
distribution. The incorporation of hypergraphs and variational inference allows
our model to capture complex, n-ary interactions among textual and visual
contents. Experimental results demonstrate that our proposed method achieves
state-of-the-art performance on the text-video retrieval task.

我们提出了一种基于分块匹配的文本 - 视频检索方法，通过构建多模态超图和引入变分推断，实现在高阶语义空间中对文本和视频的复杂多元交互进行建模，进而提高检索性能。