The goal of few-shot relation extraction is to predict relations between name
entities in a sentence when only a few labeled instances are available for
training. Existing few-shot relation extraction methods focus on uni-modal
information such as text only. This reduces performance when there are no clear
contexts between the name entities described in text. We propose a multi-modal
few-shot relation extraction model (MFS-HVE) that leverages both textual and
visual semantic information to learn a multi-modal representation jointly. The
MFS-HVE includes semantic feature extractors and multi-modal fusion components.
The MFS-HVE semantic feature extractors are developed to extract both textual
and visual features. The visual features include global image features and
local object features within the image. The MFS-HVE multi-modal fusion unit
integrates information from various modalities using image-guided attention,
object-guided attention, and hybrid feature attention to fully capture the
semantic interaction between visual regions of images and relevant texts.
Extensive experiments conducted on two public datasets demonstrate that
semantic visual information significantly improves the performance of few-shot
relation prediction.

多模态表示学习在少样本关系抽取上的应用