The task of multimodal relation extraction has attracted significant research attention, but progress is constrained by the scarcity of available training data. One natural thought is to extend existing datasets with cross-modal generative models. In this paper, we consider a novel problem setting, where only unimodal data, either text or image, are available during training. We aim to train a multimodal classifier from synthetic data that perform well on real multimodal test data. However, training with synthetic data suffers from two obstacles: lack of data diversity and label information loss. To alleviate the issues, we propose Mutual Information-aware Multimodal Iterated Relational dAta GEneration (MI2RAGE), which applies Chained Cross-modal Generation (CCG) to promote diversity in the generated data and exploits a teacher network to select valuable training samples with high mutual information with the ground-truth labels. Comparing our method to direct training on synthetic data, we observed a significant improvement of 24.06% F1 with synthetic text and 26.42% F1 with synthetic images. Notably, our best model trained on completely synthetic images outperforms prior state-of-the-art models trained on real multimodal data by a margin of 3.76% in F1. Our codebase will be made available upon acceptance.

使用 Mutual Information-aware Multimodal Iterated Relational dAta GEneration (MI2RAGE) 方法，通过 Chained Cross-modal Generation (CCG) 促进生成数据的多样性，并利用 teacher network 选择与 ground-truth 标签具有高互信息的有价值的训练样本，相比于直接在合成数据上训练，我们观察到合成文本的 F1 得分提高了 24.06%，合成图像的 F1 得分提高了 26.42%。值得注意的是，我们最好的模型在完全合成图像上训练时，相对于在真实多模态数据上训练的现有最先进模型，在 F1 得分上提升了 3.76%。

合成数据训练在多模态关系抽取中胜过真实数据