We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP
employs two acoustic representation models, EnCodec and CLAP, along with a
pretrained language model, BART. We also introduce a new training objective
called masked codec modeling that improves acoustic awareness of the pretrained
language model. Experimental results on AudioCaps and Clotho demonstrate that
our model surpasses the performance of baseline models. Source code will be
available at this https URL . An online demo is
available at this https URL .

我们提出了一种自动音频字幕的新框架 EnCLAP，使用了两个声学表示模型 EnCodec 和 CLAP 以及一个预训练语言模型 BART。我们还引入了一种称为 masked codec modeling 的新训练目标，提高了预训练语言模型的声学感知能力。在 AudioCaps 和 Clotho 上的实验结果表明，我们的模型超过了基准模型的性能。我们将在此网址上提供源代码。在线演示可在此网址上获得。

EnCLAP：基于神经音频编解码器和音频文本联合嵌入的自动音频字幕生成

EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for  Automated Audio Captioning

The goal of universal audio representation learning is to obtain foundational
models that can be used for a variety of downstream tasks involving speech,
music or environmental sounds. To approach this problem, methods inspired by
self-supervised models from NLP, like BERT, are often used and adapted to
audio. These models rely on the discrete nature of text, hence adopting this
type of approach for audio processing requires either a change in the learning
objective or mapping the audio signal to a set of discrete classes. In this
work, we explore the use of EnCodec, a neural audio codec, to generate discrete
targets for learning an universal audio model based on a masked autoencoder
(MAE). We evaluate this approach, which we call EncodecMAE, on a wide range of
audio tasks spanning speech, music and environmental sounds, achieving
performances comparable or better than leading audio representation models.

本研究探讨了使用神经音频编解码器 EnCodec 生成发音、音乐和环境声音的离散目标，以学习基于遮蔽自动编码器的通用音频模型 EncodecMAE，并在广泛的音频任务中取得了与领先的音频表示模型相媲美或更好的性能。

EnCodecMAE: 利用神经编解码器进行通用音频表示学习

EnCodecMAE: Leveraging neural codecs for universal audio representation  learning

This paper presents FunCodec, a fundamental neural speech codec toolkit,
which is an extension of the open-source speech processing toolkit FunASR.
FunCodec provides reproducible training recipes and inference scripts for the
latest neural speech codec models, such as SoundStream and Encodec. Thanks to
the unified design with FunASR, FunCodec can be easily integrated into
downstream tasks, such as speech recognition. Along with FunCodec, pre-trained
models are also provided, which can be used for academic or generalized
purposes. Based on the toolkit, we further propose the frequency-domain codec
models, FreqCodec, which can achieve comparable speech quality with much lower
computation and parameter complexity. Experimental results show that, under the
same compression ratio, FunCodec can achieve better reconstruction quality
compared with other toolkits and released models. We also demonstrate that the
pre-trained models are suitable for downstream tasks, including automatic
speech recognition and personalized text-to-speech synthesis. This toolkit is
publicly available at this https URL

FunCodec 是一个基于开源语音处理工具 FunASR 的基础神经语音编解码器工具包，提供训练可重现、推理脚本用于最新的神经语音编解码器模型，如 SoundStream 和 Encodec，并且可以轻松集成到下游任务中，如语音识别。其中，还提供预训练模型，可用于学术或一般用途。此工具包还进一步提出了频域编解码器模型 FreqCodec，与其他工具包和发布模型相比，在相同的压缩比下，FunCodec 可以实现更好的重构质量，并且证明预训练模型适用于包括自动语音识别和个性化文本转语音合成在内的下游任务。