As the scale of generative models continues to grow, efficient reuse and
adaptation of pre-trained models have become crucial considerations. In this
work, we propose Voicebox Adapter, a novel approach that integrates
fine-grained conditions into a pre-trained Voicebox speech generation model
using a cross-attention module. To ensure a smooth integration of newly added
modules with pre-trained ones, we explore various efficient fine-tuning
approaches. Our experiment shows that the LoRA with bias-tuning configuration
yields the best performance, enhancing controllability without compromising
speech quality. Across three fine-grained conditional generation tasks, we
demonstrate the effectiveness and resource efficiency of Voicebox Adapter.
Follow-up experiments further highlight the robustness of Voicebox Adapter
across diverse data setups.

提出了 Voicebox Adapter 方法，通过交叉注意力模块将细粒度条件整合到预训练的 Voicebox 语音生成模型中，探索了各种高效的微调方法，实验证明，采用具有偏差微调配置的 LoRA 方法性能最佳，提高了可控性同时保持了语音质量，表明 Voicebox Adapter 在三种细粒度条件生成任务中具有效力和资源效率，并且通过后续实验证明了在不同数据设置中 Voicebox Adapter 的鲁棒性。