Controllable music generation methods are critical for human-centered
AI-based music creation, but are currently limited by speed, quality, and
control design trade-offs. Diffusion Inference-Time T-optimization (DITTO), in
particular, offers state-of-the-art results, but is over 10x slower than
real-time, limiting practical use. We propose Distilled Diffusion
Inference-Time T -Optimization (or DITTO-2), a new method to speed up
inference-time optimization-based control and unlock faster-than-real-time
generation for a wide-variety of applications such as music inpainting,
outpainting, intensity, melody, and musical structure control. Our method works
by (1) distilling a pre-trained diffusion model for fast sampling via an
efficient, modified consistency or consistency trajectory distillation process
(2) performing inference-time optimization using our distilled model with
one-step sampling as an efficient surrogate optimization task and (3) running a
final multi-step sampling generation (decoding) using our estimated noise
latents for best-quality, fast, controllable generation. Through thorough
evaluation, we find our method not only speeds up generation over 10-20x, but
simultaneously improves control adherence and generation quality all at once.
Furthermore, we apply our approach to a new application of maximizing text
adherence (CLAP score) and show we can convert an unconditional diffusion model
without text inputs into a model that yields state-of-the-art text control.
Sound examples can be found at this https URL

提出了 Distilled Diffusion Inference-Time T-Optimization（DITTO-2）方法，以实现快于实时的可控音乐生成，包括音乐修补、扩展、强度、旋律和音乐结构控制，并将该方法应用于最大化文本一致性的新应用。

DITTO-2: 音乐生成的蒸馏漫扩推理时间优化

DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music  Generation

Controllable music generation plays a vital role in human-AI music
co-creation. While Large Language Models (LLMs) have shown promise in
generating high-quality music, their focus on autoregressive generation limits
their utility in music editing tasks. To bridge this gap, we introduce a novel
Parameter-Efficient Fine-Tuning (PEFT) method. This approach enables
autoregressive language models to seamlessly address music inpainting tasks.
Additionally, our PEFT method integrates frame-level content-based controls,
facilitating track-conditioned music refinement and score-conditioned music
arrangement. We apply this method to fine-tune MusicGen, a leading
autoregressive music generation model. Our experiments demonstrate promising
results across multiple music editing tasks, offering more flexible controls
for future AI-driven music editing tools. A demo
page\footnote{\url{this https URL}.} showcasing our work and
source codes\footnote{\url{this https URL}.} are available
online.

可控音乐生成对于人工智能与人类共创音乐至关重要。大型语言模型在生成高质量音乐方面表现出了潜力，但其自回归生成的特性限制了其在音乐编辑任务中的实用性。为了弥合这一差距，我们引入了一种新颖的参数高效微调（PEFT）方法，通过此方法，自回归语言模型可以无缝地解决音乐修复任务。此外，我们的 PEFT 方法结合了基于帧级内容的控制，便于音轨调节的音乐细化和谱面调节的音乐编排。我们将此方法应用于优化 MusicGen，一种领先的自回归音乐生成模型。我们的实验在多个音乐编辑任务中显示了有希望的结果，为未来的 AI 驱动音乐编辑工具提供了更灵活的控制。我们的工作演示页面和源代码可以在线上获取。

通过基于内容的控制，进行可定向的长期音乐音频生成和编辑

Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation  and Editing via Content-based Controls

We propose Polyffusion, a diffusion model that generates polyphonic music
scores by regarding music as image-like piano roll representations. The model
is capable of controllable music generation with two paradigms: internal
control and external control. Internal control refers to the process in which
users pre-define a part of the music and then let the model infill the rest,
similar to the task of masked music generation (or music inpainting). External
control conditions the model with external yet related information, such as
chord, texture, or other features, via the cross-attention mechanism. We show
that by using internal and external controls, Polyffusion unifies a wide range
of music creation tasks, including melody generation given accompaniment,
accompaniment generation given melody, arbitrary music segment inpainting, and
music arrangement given chords or textures. Experimental results show that our
model significantly outperforms existing Transformer and sampling-based
baselines, and using pre-trained disentangled representations as external
conditions yields more effective controls.

我们提出了 Polyffusion，它是一种扩散模型，通过将音乐视为图像样的钢琴卷轴表示生成重音乐谱。该模型可以通过内部控制和外部控制两种范式对音乐进行可控制生成。实验结果表明，我们的模型明显优于现有的 Transformer 和基于采样的基线，并使用预训练的解耦表示作为外部条件可以产生更有效的控制。