Pre-trained language models (PLMs) have played an increasing role in
multimedia research. In terms of vision-language (VL) tasks, they often serve
as a language encoder and still require an additional fusion network for VL
reasoning, resulting in excessive memory overhead. In this paper, we focus on
exploring PLMs as a stand-alone model for VL reasoning tasks. Inspired by the
recently popular prompt tuning, we first prove that the processed visual
features can be also projected onto the semantic space of PLMs and act as
prompt tokens to bridge the gap between single- and multi-modal learning.
However, this solution exhibits obvious redundancy in visual information and
model inference, and the placement of prompt tokens also greatly affects the
final performance. Based on these observations, we further propose a novel
transfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP).
Concretely, DVP first deploys a cross-attention module to obtain text-related
and compact visual prompt tokens, thereby greatly reducing the input length of
PLMs. To obtain the optimal placement, we also equip DVP with a
reinforcement-learning based search algorithm, which can automatically merge
DVP with PLMs for different VL tasks via a very short search process. In
addition, we also experiment DVP with the recently popular adapter approach to
keep the most parameters of PLMs intact when adapting to VL tasks, helping PLMs
achieve a quick shift between single- and multi-modal tasks. We apply DVP to
two representative PLMs, namely BERT and T5, and conduct extensive experiments
on a set of VL reasoning benchmarks including VQA2.0, GQA and SNLIVE. The
experimental results not only show the advantage of DVP on efficiency and
performance, but also confirm its superiority in adapting pre-trained language
models to VL tasks.

本文旨在研究使用预训练语言模型（Pre-trained language models，PLMs）作为一个独立的模型来进行视觉语言推理任务。提出了一个名为 Dynamic Visual Prompting（DVP）的新型转移学习方法，通过搜索算法，在保持 PLMs 参数完整性的情况下，有效地将其与不同的视觉语言任务相结合。经过广泛的实验验证，DVP 在效率和性能方面都具有优势，并且能够很好地适应预训练语言模型与 VL 任务相结合的需求。

通过动态视觉提示将预训练语言模型应用于视觉语言任务

Adapting Pre-trained Language Models to Vision-Language Tasks via  Dynamic Visual Prompting

The remarkable success of large language models has been driven by dense
models trained on massive unlabeled, unstructured corpora. These corpora
typically contain text from diverse, heterogeneous sources, but information
about the source of the text is rarely used during training. Transferring their
knowledge to a target domain is typically done by continuing training
in-domain. In this paper, we introduce a method to permit domain adaptation to
many diverse domains using a computationally efficient adapter approach. Our
method is based on the observation that textual domains are partially
overlapping, and we represent domains as a hierarchical tree structure where
each node in the tree is associated with a set of adapter weights. When
combined with a frozen pretrained language model, this approach enables
parameter sharing among related domains, while avoiding negative interference
between unrelated ones. Experimental results with GPT-2 and a large fraction of
the 100 most represented websites in C4 show across-the-board improvements
in-domain. We additionally provide an inference time algorithm for a held-out
domain and show that averaging over multiple paths through the tree enables
further gains in generalization, while adding only a marginal cost to
inference.

本文提出了一种使用计算有效的适配器方法在多个不同领域适应的方法，其通过冻结预先训练的语言模型并与每个节点相关联的适配器权重组合，实现相关领域之间的参数共享。GPT-2 和 C4 中最多的 100 个网站上的实验结果表明，这种方法可以全面提高领域内性能，并提供了一种推理时间算法来推出新领域的泛化性能，同时只能增加成本。