Since visual perception can give rich information beyond text descriptions
for world understanding, there has been increasing interest in leveraging
visual grounding for language learning. Recently, vokenization (Tan and Bansal,
2020) has attracted attention by using the predictions of a text-to-image
retrieval model as labels for language model supervision. Despite its success,
the method suffers from approximation error of using finite image labels and
the lack of vocabulary diversity of a small image-text dataset. To overcome
these limitations, we present VidLanKD, a video-language knowledge distillation
method for improving language understanding. We train a multi-modal teacher
model on a video-text dataset, and then transfer its knowledge to a student
language model with a text dataset. To avoid approximation error, we propose to
use different knowledge distillation objectives. In addition, the use of a
large-scale video-text dataset helps learn diverse and richer vocabularies. In
our experiments, VidLanKD achieves consistent improvements over text-only
language models and vokenization models, on several downstream language
understanding tasks including GLUE, SQuAD, and SWAG. We also demonstrate the
improved world knowledge, physical reasoning, and temporal reasoning
capabilities of our model by evaluating on the GLUE-diagnostics, PIQA, and
TRACIE datasets. Lastly, we present comprehensive ablation studies as well as
visualizations of the learned text-to-video grounding results of our teacher
and student language models. Our code and models are available at:
this https URL

本文提出了一种名为 VidLanKD 的视频 - 语言知识蒸馏方法，通过在视频文本数据集上训练一个多模教师模型，再将其知识转移到一个文本数据集上的学生语言模型，以达到优化语言理解的目的。实验结果表明，VidLanKD 模型在多个语言理解任务上均取得了较好的效果。

VidLanKD: 通过视频提炼的知识转移来改进语言理解

VidLanKD: Improving Language Understanding via Video-Distilled Knowledge  Transfer

Humans learn language by listening, speaking, writing, reading, and also, via
interaction with the multimodal real world. Existing language pre-training
frameworks show the effectiveness of text-only self-supervision while we
explore the idea of a visually-supervised language model in this paper. We find
that the main reason hindering this exploration is the large divergence in
magnitude and distributions between the visually-grounded language datasets and
pure-language corpora. Therefore, we develop a technique named "vokenization"
that extrapolates multimodal alignments to language-only data by contextually
mapping language tokens to their related images (which we call "vokens"). The
"vokenizer" is trained on relatively small image captioning datasets and we
then apply it to generate vokens for large language corpora. Trained with these
contextually generated vokens, our visually-supervised language models show
consistent improvements over self-supervised alternatives on multiple
pure-language tasks such as GLUE, SQuAD, and SWAG. Code and pre-trained models
publicly available at this https URL

本研究提出了「vokenization」技术，通过将语言令牌与相关的图像（称为「vokens」）上下文映射到语言仅数据中，将多模态对齐推广到仅语言数据，并利用这些生成的 vokens 进行训练。使用这些训练有素的 vokens，我们的视觉监督语言模型在多个语言任务上均表现出了一致的改进。