Multimodal learning pipelines have benefited from the success of pretrained
language models. However, this comes at the cost of increased model parameters.
In this work, we propose Adapted Multimodal BERT (AMB), a BERT-based
architecture for multimodal tasks that uses a combination of adapter modules
and intermediate fusion layers. The adapter adjusts the pretrained language
model for the task at hand, while the fusion layers perform task-specific,
layer-wise fusion of audio-visual information with textual BERT
representations. During the adaptation process the pre-trained language model
parameters remain frozen, allowing for fast, parameter-efficient training. In
our ablations we see that this approach leads to efficient models, that can
outperform their fine-tuned counterparts and are robust to input noise. Our
experiments on sentiment analysis with CMU-MOSEI show that AMB outperforms the
current state-of-the-art across metrics, with 3.4% relative reduction in the
resulting error and 2.1% relative improvement in 7-class classification
accuracy.

本文通过适配器模块与中间融合层来调整预训练语言模型，从而建立了适应多模态任务的基于 BERT 的体系结构 AMB，并在情感分析方面取得 3.4% 的误差相对降低率和 2.1% 的 7 级分类精度相对提高率的最好表现，展现了 AMB 的高效性与稳健性。