We propose a modular framework that leverages the expertise of different
foundation models over different modalities and domains in order to perform a
single, complex, multi-modal task, without relying on prompt engineering or
otherwise tailor-made multi-modal training. Our approach enables decentralized
command execution and allows each model to both contribute and benefit from the
expertise of the other models. Our method can be extended to a variety of
foundation models (including audio and vision), above and beyond only language
models, as it does not depend on prompts. We demonstrate our approach on two
tasks. On the well-known task of stylized image captioning, our experiments
show that our approach outperforms semi-supervised state-of-the-art models,
while being zero-shot and avoiding costly training, data collection, and prompt
engineering. We further demonstrate this method on a novel task, audio-aware
image captioning, in which an image and audio are given and the task is to
generate text that describes the image within the context of the provided
audio. Our code is available on GitHub.

我们提出了一个模块化框架，利用不同模态和领域中不同基础模型的专业知识，以执行一个单一、复杂的多模态任务，无需依赖提示工程或量身定制的多模态训练。我们的方法实现了分散的命令执行，使每个模型能够同时贡献和受益于其他模型的专业知识。我们的方法可以扩展到各种基础模型（包括音频和视觉），不仅限于语言模型，因为它不依赖于提示。我们在两个任务上演示了我们的方法。在众所周知的样式化图像字幕任务中，我们的实验表明我们的方法优于半监督最先进模型，同时具备零 - shot 能力，避免了昂贵的训练、数据收集和提示工程。我们进一步在一个新颖的任务上演示了这个方法，即音频感知图像字幕，在这个任务中，给定图像和音频，任务是在提供的音频语境中生成描述图像的文本。我们的代码可在 GitHub 上找到。

Apollo：多模式无需标注的零 - shot 推理与多专家

Apollo: Zero-shot MultiModal Reasoning with Multiple Experts

Generating visually grounded image captions with specific linguistic styles
using unpaired stylistic corpora is a challenging task, especially since we
expect stylized captions with a wide variety of stylistic patterns. In this
paper, we propose a novel framework to generate Accurate and Diverse Stylized
Captions (ADS-Cap). Our ADS-Cap first uses a contrastive learning module to
align the image and text features, which unifies paired factual and unpaired
stylistic corpora during the training process. A conditional variational
auto-encoder is then used to automatically memorize diverse stylistic patterns
in latent space and enhance diversity through sampling. We also design a simple
but effective recheck module to boost style accuracy by filtering
style-specific captions. Experimental results on two widely used stylized image
captioning datasets show that regarding consistency with the image, style
accuracy and diversity, ADS-Cap achieves outstanding performances compared to
various baselines. We finally conduct extensive analyses to understand the
effectiveness of our method. Our code is available at
this https URL

使用不成对的文体语料库生成具有特定语言风格的视觉 grounded 图片标题是一项具有挑战性的任务，本文提出了一种新的框架（ADS-Cap）来生成准确且多样化的文体标题，其中使用对比学习模块来统一成对的事实语料库和不成对的文体语料库，并使用条件变分自动编码器在潜空间中自动记忆多种文体模式以增强多样性，并通过简单而有效的重新检查模块提高文体准确性。在两个广泛使用的文体图片标题数据集上进行的实验结果表明，相比多种基准模型，ADS-Cap 在与图片一致性、文体准确性和多样性方面表现出了出色的性能。最后，我们进行了广泛的分析以了解我们方法的有效性。

ADS-Cap: 一个准确且多样化的风格化说明文档生成框架，无需成对的风格化语料库

ADS-Cap: A Framework for Accurate and Diverse Stylized Captioning with  Unpaired Stylistic Corpora

Generating stylized captions for an image is an emerging topic in image
captioning. Given an image as input, it requires the system to generate a
caption that has a specific style (e.g., humorous, romantic, positive, and
negative) while describing the image content semantically accurately. In this
paper, we propose a novel stylized image captioning model that effectively
takes both requirements into consideration. To this end, we first devise a new
variant of LSTM, named style-factual LSTM, as the building block of our model.
It uses two groups of matrices to capture the factual and stylized knowledge,
respectively, and automatically learns the word-level weights of the two groups
based on previous context. In addition, when we train the model to capture
stylized elements, we propose an adaptive learning approach based on a
reference factual model, it provides factual knowledge to the model as the
model learns from stylized caption labels, and can adaptively compute how much
information to supply at each time step. We evaluate our model on two stylized
image captioning datasets, which contain humorous/romantic captions and
positive/negative captions, respectively. Experiments shows that our proposed
model outperforms the state-of-the-art approaches, without using extra ground
truth supervision.

本文提出了一种新颖的基于 style-factual LSTM 和自适应学习的图像标注模型，能够同时考虑图像的内容和特定的风格要求，如幽默，浪漫，正面和负面。实验结果表明，该模型在两个不同的图像标注数据集上（幽默 / 浪漫标注以及正面 / 负面标注）的表现均超过了目前的最先进方法，且不需要额外的监督信息。