We propose a modular framework that leverages the expertise of different
foundation models over different modalities and domains in order to perform a
single, complex, multi-modal task, without relying on prompt engineering or
otherwise tailor-made multi-modal training. Our approach enables decentralized
command execution and allows each model to both contribute and benefit from the
expertise of the other models. Our method can be extended to a variety of
foundation models (including audio and vision), above and beyond only language
models, as it does not depend on prompts. We demonstrate our approach on two
tasks. On the well-known task of stylized image captioning, our experiments
show that our approach outperforms semi-supervised state-of-the-art models,
while being zero-shot and avoiding costly training, data collection, and prompt
engineering. We further demonstrate this method on a novel task, audio-aware
image captioning, in which an image and audio are given and the task is to
generate text that describes the image within the context of the provided
audio. Our code is available on GitHub.

我们提出了一个模块化框架，利用不同模态和领域中不同基础模型的专业知识，以执行一个单一、复杂的多模态任务，无需依赖提示工程或量身定制的多模态训练。我们的方法实现了分散的命令执行，使每个模型能够同时贡献和受益于其他模型的专业知识。我们的方法可以扩展到各种基础模型（包括音频和视觉），不仅限于语言模型，因为它不依赖于提示。我们在两个任务上演示了我们的方法。在众所周知的样式化图像字幕任务中，我们的实验表明我们的方法优于半监督最先进模型，同时具备零 - shot 能力，避免了昂贵的训练、数据收集和提示工程。我们进一步在一个新颖的任务上演示了这个方法，即音频感知图像字幕，在这个任务中，给定图像和音频，任务是在提供的音频语境中生成描述图像的文本。我们的代码可在 GitHub 上找到。