The task of audio captioning is similar in essence to tasks such as image and
video captioning. However, it has received much less attention. We propose
three desiderata for captioning audio -- (i) fluency of the generated text,
(ii) faithfulness of the generated text to the input audio, and the somewhat
related (iii) audibility, which is the quality of being able to be perceived
based only on audio. Our method is a zero-shot method, i.e., we do not learn to
perform captioning. Instead, captioning occurs as an inference process that
involves three networks that correspond to the three desired qualities: (i) A
Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A
model that provides a matching score between an audio file and a text, for
which we use a multimodal matching network called ImageBind, and (iii) A text
classifier, trained using a dataset we collected automatically by instructing
GPT-4 with prompts designed to direct the generation of both audible and
inaudible sentences. We present our results on the AudioCap dataset,
demonstrating that audibility guidance significantly enhances performance
compared to the baseline, which lacks this objective.

音频字幕任务与图像和视频字幕任务在本质上相似，但鲜有关注。我们提出了音频字幕的三个愿景：生成文本的流畅性，生成文本对输入音频的忠实度以及听觉可感知性。我们的方法是零样本方法，不通过学习而是通过推理过程来进行字幕生成，其中涉及的三个网络对应三个期望的质量：（i）一个大型语言模型，本文中为 GPT-2，（ii）一个在音频文件和文本之间提供匹配分数的模型，我们使用一种名为 ImageBind 的多模态匹配网络，（iii）一个文本分类器，利用我们自动收集的数据集进行训练，数据集是通过指导 GPT-4 生成可听和不可听的句子而设计的。我们在 AudioCap 数据集上展示了我们的结果，证明了听觉指导明显提高了性能，相比缺乏这一目标的基准系统。