The abilities of large language models (LLMs) have recently progressed to
unprecedented levels, paving the way to novel applications in a wide variety of
areas. In computer vision, LLMs can be used to prime vision-language tasks such
image captioning and visual question answering when coupled with pre-trained
vision backbones. While different approaches have been explored to interface
LLMs with ``perceptual backbones'' that process, e.g., visual or audio data,
they are often explored for different tasks, different datasets, and using
different perceptual backbones and language models, hindering direct comparison
of the interfacing mechanisms. To remedy this lack of comparability between
methods, we present an extensive experimental evaluation of different
interfacing mechanisms, across multiple tasks (including image, video, and
audio captioning as well as visual question answering), datasets and backbones,
paying special attention to low-data settings. We find improved performance
using existing mechanisms over state-of-the-art results, and identify a new
interfacing mechanism that yields (near) optimal results across different
tasks, while obtaining a 4x reduction in training time.

大语言模型在计算机视觉领域中通过不同的接口机制实现图像字幕和视觉问题回答的任务，通过实验评估各种接口机制和数据集，发现现有机制在多个任务中表现更好，并识别出一种新的接口机制，在不同任务上获得接近最优的结果，并降低了训练时间。