The ability to quickly learn from a small quantity oftraining data widens the
range of machine learning applications. In this paper, we propose a
data-efficient image captioning model, VisualGPT, which leverages the
linguistic knowledge from a large pretrained language model(LM). A crucial
challenge is to balance between the use of visual information in the image and
prior linguistic knowledge acquired from pretraining. We designed a novel
self-resurrecting encoder-decoder attention mechanism to quickly adapt the
pretrained LM as the language decoder ona small amount of in-domain training
data. The proposed self-resurrecting activation unit produces sparse
activations but has reduced susceptibility to zero gradients. We train the
proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual
Captions training data. Under these conditions, we outperform the best baseline
model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual
Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray,
a medical report generation dataset. To the best of our knowledge, this is the
first work that improves data efficiency of image captioning by utilizing LM
pretrained on unimodal data. Our code is available at:
this https URL

本研究提出了 VisualGPT，一种数据高效的图像字幕模型，它利用了预先训练的语言模型中获得的语言知识，使用自重生编码器 - 解码器注意机制在少量领域训练数据上快速适应预训练的语言模型，并通过稀疏激活单元减少了零梯度的影响，我们在 MSCOCO 和 Conceptual Captions 数据集上进行 0.1％，0.5％和 1％的训练，结果表明，我们在 MS COCO 上的 CIDEr 得分最好的基线模型高达 10.8％，在 Conceptual Captions 上高达 5.4％，并在医学报告生成数据集 IU X-ray 上取得了最新的结果。