Amid the rising intersection of generative AI and human artistic processes,
this study probes the critical yet less-explored terrain of alignment in
human-centric automatic song composition. We propose a novel task of Colloquial
Description-to-Song Generation, which focuses on aligning the generated content
with colloquial human expressions. This task is aimed at bridging the gap
between colloquial language understanding and auditory expression within an AI
model, with the ultimate goal of creating songs that accurately satisfy human
auditory expectations and structurally align with musical norms. Current
datasets are limited due to their narrow descriptive scope, semantic gaps and
inaccuracies. To overcome data scarcity in this domain, we present the Caichong
Music Dataset (CaiMD). CaiMD is manually annotated by both professional
musicians and amateurs, offering diverse perspectives and a comprehensive
understanding of colloquial descriptions. Unlike existing datasets pre-set with
expert annotations or auto-generated ones with inherent biases, CaiMD caters
more sufficiently to our purpose of aligning AI-generated music with widespread
user-desired results. Moreover, we propose an innovative single-stage framework
called MuDiT/MuSiT for enabling effective human-machine alignment in song
creation. This framework not only achieves cross-modal comprehension between
colloquial language and auditory music perceptions but also ensures generated
songs align with user-desired results. MuDiT/MuSiT employs one DiT/SiT model
for end-to-end generation of musical components like melody, harmony, rhythm,
vocals, and instrumentation. The approach ensures harmonious sonic cohesiveness
amongst all generated musical components, facilitating better resonance with
human auditory expectations.

该研究探索生成式人工智能与人类艺术过程相交的关键领域，针对人类为中心的自动歌曲创作中对齐的关系进行研究。通过提出口语描述到歌曲生成的新任务，着重于将生成的内容与口语人类表达相对齐。为了充分满足人类听觉期望并与音乐规范结构对齐的最终目标，该任务旨在弥合口语语言理解和 AI 模型中的听觉表达之间的鸿沟。通过介绍由专业音乐家和业余爱好者手动注释的 Caichong 音乐数据集（CaiMD），克服了该领域中数据匮乏的限制。与现有具有专家注释或存在固有偏见的自动生成数据集不同，CaiMD 更能满足我们将 AI 生成的音乐与用户期望结果对齐的目的。此外，提出了一种创新的名为 MuDiT/MuSiT 的单阶段框架，用于在歌曲创作中实现有效的人机对齐。该框架不仅实现了口语语言与音乐知觉之间的跨模态理解，还确保生成的歌曲与用户期望结果对齐。MuDiT/MuSiT 采用一种 DiT/SiT 模型，用于端到端生成如旋律、和声、节奏、人声和乐器等音乐元素。该方法确保所有生成的音乐元素之间具有和谐的音响连贯性，以更好地与人类听觉期望产生共鸣。