The speech-to-singing (STS) voice conversion task aims to generate singing
samples corresponding to speech recordings while facing a major challenge: the
alignment between the target (singing) pitch contour and the source (speech)
content is difficult to learn in a text-free situation. This paper proposes
AlignSTS, an STS model based on explicit cross-modal alignment, which views
speech variance such as pitch and content as different modalities. Inspired by
the mechanism of how humans will sing the lyrics to the melody, AlignSTS: 1)
adopts a novel rhythm adaptor to predict the target rhythm representation to
bridge the modality gap between content and pitch, where the rhythm
representation is computed in a simple yet effective way and is quantized into
a discrete space; and 2) uses the predicted rhythm representation to re-align
the content based on cross-attention and conducts a cross-modal fusion for
re-synthesize. Extensive experiments show that AlignSTS achieves superior
performance in terms of both objective and subjective metrics. Audio samples
are available at this https URL

该研究提出了基于明确的跨模态对齐的 STS 模型 AlignSTS，用于进行语音到歌唱的转换，实验结果表明，在客观和主观指标上 AlignSTS 表现优异。