Multimodal information extraction is attracting research attention nowadays,
which requires aggregating representations from different modalities. In this
paper, we present the Intra- and Inter-Sample Relationship Modeling (I2SRM)
method for this task, which contains two modules. Firstly, the intra-sample
relationship modeling module operates on a single sample and aims to learn
effective representations. Embeddings from textual and visual modalities are
shifted to bridge the modality gap caused by distinct pre-trained language and
image models. Secondly, the inter-sample relationship modeling module considers
relationships among multiple samples and focuses on capturing the interactions.
An AttnMixup strategy is proposed, which not only enables collaboration among
samples but also augments data to improve generalization. We conduct extensive
experiments on the multimodal named entity recognition datasets Twitter-2015
and Twitter-2017, and the multimodal relation extraction dataset MNRE. Our
proposed method I2SRM achieves competitive results, 77.12% F1-score on
Twitter-2015, 88.40% F1-score on Twitter-2017, and 84.12% F1-score on MNRE.

该论文介绍了用于多模态信息提取的 Intra- and Inter-Sample Relationship Modeling (I2SRM) 方法，其中包含两个模块：内部样本关系建模模块和样本之间关系建模模块。通过在单个样本上操作来学习有效的表示，并提出了 AttnMixup 策略来改善泛化性能，在多模态实体识别数据集和多模态关系抽取数据集上进行了广泛实验，表明该方法取得了竞争性的效果。