Existing research on multimodal relation extraction (MRE) faces two
co-existing challenges, internal-information over-utilization and
external-information under-exploitation. To combat that, we propose a novel
framework that simultaneously implements the idea of internal-information
screening and external-information exploiting. First, we represent the
fine-grained semantic structures of the input image and text with the visual
and textual scene graphs, which are further fused into a unified cross-modal
graph (CMG). Based on CMG, we perform structure refinement with the guidance of
the graph information bottleneck principle, actively denoising the
less-informative features. Next, we perform topic modeling over the input image
and text, incorporating latent multimodal topic features to enrich the
contexts. On the benchmark MRE dataset, our system outperforms the current best
model significantly. With further in-depth analyses, we reveal the great
potential of our method for the MRE task. Our codes are open at
this https URL

我们提出了一种新颖的框架，该框架同时实现了内部信息筛选和外部信息利用的想法，通过对图形信息瓶颈原则的引导进行结构细化，将视觉和文本场景图进一步融合成统一的跨模式图形 (CMG)，通过加入潜在多模态主题特征来丰富语境，在基准 MRE 数据集上，我们的系统显着优于当前最佳模型。