Current multimodal models, aimed at solving Vision and Language (V+L) tasks,
predominantly repurpose Vision Encoders (VE) as feature extractors. While many
VEs -- of different architectures, trained on different data and objectives --
are publicly available, they are not designed for the downstream V+L tasks.
Nonetheless, most current work assumes that a \textit{single} pre-trained VE
can serve as a general-purpose encoder. In this work, we evaluate whether the
information stored within different VEs is complementary, i.e. if providing the
model with features from multiple VEs can improve the performance on a target
task. We exhaustively experiment with three popular VEs on six downstream V+L
tasks and analyze the attention and VE-dropout patterns. Our results and
analyses suggest that diverse VEs complement each other, resulting in improved
downstream V+L task performance, where the improvements are not due to simple
ensemble effects (i.e. the performance does not always improve when increasing
the number of encoders). We demonstrate that future VEs, which are not
\textit{repurposed}, but explicitly \textit{designed} for V+L tasks, have the
potential of improving performance on the target V+L tasks.

本研究利用三个常用的视觉编码器对六种下游视觉语言任务进行了详细实验，并对注意力机制和编码器 - dropout 模式进行了分析，结果显示不同的视觉编码器互补，可以提高下游视觉语言任务的性能而不是简单的合成效果，且未来的视觉编码器有望提高目标视觉语言任务的性能。