Multimodal named entity recognition and relation extraction (MNER and MRE) is
a fundamental and crucial branch in information extraction. However, existing
approaches for MNER and MRE usually suffer from error sensitivity when
irrelevant object images incorporated in texts. To deal with these issues, we
propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for
visual-enhanced entity and relation extraction, aiming to achieve more
effective and robust performance. Specifically, we regard visual representation
as pluggable visual prefix to guide the textual representation for error
insensitive forecasting decision. We further propose a dynamic gated
aggregation strategy to achieve hierarchical multi-scaled visual features as
visual prefix for fusion. Extensive experiments on three benchmark datasets
demonstrate the effectiveness of our method, and achieve state-of-the-art
performance. Code is available in this https URL.

本文研究了基于多模态数据的命名实体识别和关系提取，提出了一种新的基于视觉增强的实体和关系提取方法，即分层视觉前缀融合网络（HVPNeT），通过使用动态门控聚合策略，利用多层次视觉特征进行视觉前缀的融合，从而实现更有效和稳健的性能，实验证明了此方法的有效性和领先性。