Large audio-video language models can generate descriptions for both video
and audio. However, they sometimes ignore audio content, producing audio
descriptions solely reliant on visual information. This paper refers to this as
audio hallucinations and analyzes them in large audio-video language models. We
gather 1,000 sentences by inquiring about audio information and annotate them
whether they contain hallucinations. If a sentence is hallucinated, we also
categorize the type of hallucination. The results reveal that 332 sentences are
hallucinated with distinct trends observed in nouns and verbs for each
hallucination type. Based on this, we tackle a task of audio hallucination
classification using pre-trained audio-text models in the zero-shot and
fine-tuning settings. Our experimental results reveal that the zero-shot models
achieve higher performance (52.2% in F1) than the random (40.3%) and the
fine-tuning models achieve 87.9%, outperforming the zero-shot models.

通过分析大型音视频语言模型中的音频幻觉，本文收集了包含幻觉的 1,000 个句子，并通过预训练的音频文本模型以零样本和微调的方式进行音频幻觉分类任务，结果显示零样本模型表现更好 (52.2% F1)，优于随机模型 (40.3%)，微调模型表现最佳 (87.9%)。