Multimodal knowledge graph link prediction aims to improve the accuracy and
efficiency of link prediction tasks for multimodal data. However, for complex
multimodal information and sparse training data, it is usually difficult to
achieve interpretability and high accuracy simultaneously for most methods. To
address this difficulty, a new model is developed in this paper, namely
Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence
Modeling (IMKGA-SM). First, a multi-modal fine-grained fusion method is
proposed, and Vgg16 and Optical Character Recognition (OCR) techniques are
adopted to effectively extract text information from images and images. Then,
the knowledge graph link prediction task is modelled as an offline
reinforcement learning Markov decision model, which is then abstracted into a
unified sequence framework. An interactive perception-based reward expectation
mechanism and a special causal masking mechanism are designed, which "converts"
the query into an inference path. Then, an autoregressive dynamic gradient
adjustment mechanism is proposed to alleviate the insufficient problem of
multimodal optimization. Finally, two datasets are adopted for experiments, and
the popular SOTA baselines are used for comparison. The results show that the
developed IMKGA-SM achieves much better performance than SOTA baselines on
multimodal link prediction datasets of different sizes.

本文提出了 IMKGA-SM 模型来解决多模态情况下，知识图谱链接预测的解释性和高准确性问题，使用多模态细粒度融合方法处理图像和文本信息，并将链接预测任务建模为离线强化学习马尔科夫决策模型，并通过交互式感知奖励预期机制和因果屏蔽机制将查询 “转换” 为推断路径，最后在两个数据集上验证实验结果显示 IMKGA-SM 相比现有的 SOTA 方法在不同大小的多模态链接预测数据集上表现更好。