In this work, we introduce a challenging task for simultaneously generating
3D holistic body motions and singing vocals directly from textual lyrics
inputs, advancing beyond existing works that typically address these two
modalities in isolation. To facilitate this, we first collect the RapVerse
dataset, a large dataset containing synchronous rapping vocals, lyrics, and
high-quality 3D holistic body meshes. With the RapVerse dataset, we investigate
the extent to which scaling autoregressive multimodal transformers across
language, audio, and motion can enhance the coherent and realistic generation
of vocals and whole-body human motions. For modality unification, a
vector-quantized variational autoencoder is employed to encode whole-body
motion sequences into discrete motion tokens, while a vocal-to-unit model is
leveraged to obtain quantized audio tokens preserving content, prosodic
information, and singer identity. By jointly performing transformer modeling on
these three modalities in a unified way, our framework ensures a seamless and
realistic blend of vocals and human motions. Extensive experiments demonstrate
that our unified generation framework not only produces coherent and realistic
singing vocals alongside human motions directly from textual inputs but also
rivals the performance of specialized single-modality generation systems,
establishing new benchmarks for joint vocal-motion generation. The project page
is available for research purposes at this https URL

通过将三维整体身体动作和唱歌声音联合生成，我们的研究工作在文本歌词输入方面取得了进展，并建立了整体人体动作和声音生成的新基准。