Text-based audio generation models have limitations as they cannot encompass
all the information in audio, leading to restricted controllability when
relying solely on text. To address this issue, we propose a novel model that
enhances the controllability of existing pre-trained text-to-audio models by
incorporating additional conditions including content (timestamp) and style
(pitch contour and energy contour) as supplements to the text. This approach
achieves fine-grained control over the temporal order, pitch, and energy of
generated audio. To preserve the diversity of generation, we employ a trainable
control condition encoder that is enhanced by a large language model and a
trainable Fusion-Net to encode and fuse the additional conditions while keeping
the weights of the pre-trained text-to-audio model frozen. Due to the lack of
suitable datasets and evaluation metrics, we consolidate existing datasets into
a new dataset comprising the audio and corresponding conditions and use a
series of evaluation metrics to evaluate the controllability performance.
Experimental results demonstrate that our model successfully achieves
fine-grained control to accomplish controllable audio generation. Audio samples
and our dataset are publicly available at
this https URL

我们提出了一种新的模型，通过包含额外条件（时间戳、语调曲线和能量曲线）作为文本的补充，增强了现有预训练文本转音频模型的可控性，实现了对生成音频的时间顺序、音高和能量的精细控制。通过使用可训练的控制条件编码器和可训练的融合网络，在保持预训练文本转音频模型权重不变的同时，将额外条件编码和融合。由于缺乏合适的数据集和评估指标，我们将现有数据集整合成一个包含音频和相应条件的新数据集，并使用一系列评估指标来评估可控性能。实验结果表明，我们的模型成功实现了细粒度控制，实现了可控的音频生成。音频样本和我们的数据集可在此 https URL 获取。