This paper studies the multimodal named entity recognition (MNER) and
multimodal relation extraction (MRE), which are important for multimedia social
platform analysis. The core of MNER and MRE lies in incorporating evident
visual information to enhance textual semantics, where two issues inherently
demand investigations. The first issue is modality-noise, where the
task-irrelevant information in each modality may be noises misleading the task
prediction. The second issue is modality-gap, where representations from
different modalities are inconsistent, preventing from building the semantic
alignment between the text and image. To address these issues, we propose a
novel method for MNER and MRE by Multi-Modal representation learning with
Information Bottleneck (MMIB). For the first issue, a refinement-regularizer
probes the information-bottleneck principle to balance the predictive evidence
and noisy information, yielding expressive representations for prediction. For
the second issue, an alignment-regularizer is proposed, where a mutual
information-based item works in a contrastive manner to regularize the
consistent text-image representations. To our best knowledge, we are the first
to explore variational IB estimation for MNER and MRE. Experiments show that
MMIB achieves the state-of-the-art performances on three public benchmarks.

文章研究了多模态命名实体识别和多模态关系提取的方法，并提出了一种基于信息瓶颈的多模态表示学习方法，该方法在三个公共基准测试上取得了最先进的性能。