This study introduces a novel training paradigm, audio difference learning,
for improving audio captioning. The fundamental concept of the proposed
learning method is to create a feature representation space that preserves the
relationship between audio, enabling the generation of captions that detail
intricate audio information. This method employs a reference audio along with
the input audio, both of which are transformed into feature representations via
a shared encoder. Captions are then generated from these differential features
to describe their differences. Furthermore, a unique technique is proposed that
involves mixing the input audio with additional audio, and using the additional
audio as a reference. This results in the difference between the mixed audio
and the reference audio reverting back to the original input audio. This allows
the original input's caption to be used as the caption for their difference,
eliminating the need for additional annotations for the differences. In the
experiments using the Clotho and ESC50 datasets, the proposed method
demonstrated an improvement in the SPIDEr score by 7% compared to conventional
methods.

该研究介绍了一种新颖的训练范式，音频差异学习，用于改进音频字幕生成。该方法通过创建一个保留音频关系的特征表示空间，能够生成详细描绘复杂音频信息的字幕。实验证明，该方法在 Clotho 和 ESC50 数据集上，相较传统方法，SPIDEr 分数提高了 7%。