Automated audio captioning (AAC) has developed rapidly in recent years,
involving acoustic signal processing and natural language processing to
generate human-readable sentences for audio clips. The current models are
generally based on the neural encoder-decoder architecture, and their decoder
mainly uses acoustic information that is extracted from the CNN-based encoder.
However, they have ignored semantic information that could help the AAC model
to generate meaningful descriptions. This paper proposes a novel approach for
automated audio captioning based on incorporating semantic and acoustic
information. Specifically, our audio captioning model consists of two
sub-modules. (1) The pre-trained keyword encoder utilizes pre-trained ResNet38
to initialize its parameters, and then it is trained by extracted keywords as
labels. (2) The multi-modal attention decoder adopts an LSTM-based decoder that
contains semantic and acoustic attention modules. Experiments demonstrate that
our proposed model achieves state-of-the-art performance on the Clotho dataset.
Our code can be found at this https URL

本文提出了一种基于语义和音频信息相结合的自动音频字幕生成模型，使用预训练的 ResNet38 来初始化预训练关键字编码器，并使用 LSTM 解码器和语义和音频注意力模块来进行多模态注意力解码，实验证明该模型在 Clotho 数据集上达到了最先进的性能。