Multimodal information extraction (MIE) gains significant attention as the
popularity of multimedia content increases. However, current MIE methods often
resort to using task-specific model structures, which results in limited
generalizability across tasks and underutilizes shared knowledge across MIE
tasks. To address these issues, we propose UMIE, a unified multimodal
information extractor to unify three MIE tasks as a generation problem using
instruction tuning, being able to effectively extract both textual and visual
mentions. Extensive experiments show that our single UMIE outperforms various
state-of-the-art (SoTA) methods across six MIE datasets on three tasks.
Furthermore, in-depth analysis demonstrates UMIE's strong generalization in the
zero-shot setting, robustness to instruction variants, and interpretability.
Our research serves as an initial step towards a unified MIE model and
initiates the exploration into both instruction tuning and large language
models within the MIE domain. Our code, data, and model are available at
this https URL

多模态信息提取通过指令调整以解决当前面临的任务特定模型结构的持续问题，提供了一个统一的多模态信息提取器 UMIE，能够有效地提取文本和视觉提及，并在六个多模态信息提取数据集上的三个任务中在零样本模型、指令变体和可解释性方面都表现出强大的泛化能力。