Image-language learning has made unprecedented progress in visual
understanding. These developments have come at high costs, as contemporary
vision-language models require large model scales and amounts of data. We here
propose a much easier recipe for image-language learning, which produces
effective models, outperforming bigger and more expensive ones, often trained
on orders of magnitude larger datasets. Our key finding is the joint learning
of a compact vision and language representation, which adaptively and
iteratively fuses the multi-modal features. This results in a more effective
image-language learning, greatly lowering the FLOPs by combining and reducing
the number of tokens for both text and images, e.g. a 33\% reduction in FLOPs
is achieved, compared to baseline fusion techniques used by popular
image-language models, while improving performance. This also allows the model
to scale without a large increase in FLOPs or memory. In addition, we propose
adaptive pre-training data sampling which improves the data efficiency. The
proposed approach achieves competitive performance compared to much larger
models, and does so with significantly less data and FLOPs. With only 40M
training examples and with 39 GFLOPs our lightweight model outperforms many
times larger state-of-the-art models of 2-20x more FLOPs and using bigger
datasets some of which with close to 1B training examples.

通过联合学习紧凑的视觉和语言表示形式以及使用自适应的、迭代融合多模态特征的方法来实现图像 - 语言学习，并增加数据效率和降低计算复杂度。该方法相比于当前流行的图像 - 语言模型而言，可降低 33% 的浮点数操作次数，同时提高性能，而只需要较少的数据和计算资源，这比使用更大的数据集和 FLOPs 几乎是 2-20 倍的最新模型更为优秀。

图像 - 语言学习的联合自适应表示

Joint Adaptive Representations for Image-Language Learning

Pre-trained language models (PLMs) have achieved remarkable success on
various natural language understanding tasks. Simple fine-tuning of PLMs, on
the other hand, might be suboptimal for domain-specific tasks because they
cannot possibly cover knowledge from all domains. While adaptive pre-training
of PLMs can help them obtain domain-specific knowledge, it requires a large
training cost. Moreover, adaptive pre-training can harm the PLM's performance
on the downstream task by causing catastrophic forgetting of its general
knowledge. To overcome such limitations of adaptive pre-training for PLM
adaption, we propose a novel domain adaption framework for PLMs coined as
Knowledge-Augmented Language model Adaptation (KALA), which modulates the
intermediate hidden representations of PLMs with domain knowledge, consisting
of entities and their relational facts. We validate the performance of our KALA
on question answering and named entity recognition tasks on multiple datasets
across various domains. The results show that, despite being computationally
efficient, our KALA largely outperforms adaptive pre-training. Code is
available at: this https URL.

本文提出了一种面向特定领域的 PLMs 领域适应框架 ——KALA（Knowledge-Augmented Language model Adaptation），通过引入实体和关系事实的领域知识来调制 PLMs 的中间隐藏表示，有效地提高了 PLMs 在不同领域的问答和命名实体识别任务上的性能，而且计算效率比自适应预训练要高。