Perceptual modification of voice is an elusive goal. While non-experts can
modify an image or sentence perceptually with available tools, it is not clear
how to similarly modify speech along perceptual axes. Voice conversion does
make it possible to convert one voice to another, but these modifications are
handled by black box models, and the specifics of what perceptual qualities to
modify and how to modify them are unclear. Towards allowing greater perceptual
control over voice, we introduce PerMod, a conditional latent diffusion model
that takes in an input voice and a perceptual qualities vector, and produces a
voice with the matching perceptual qualities. Unlike prior work, PerMod
generates a new voice corresponding to specific perceptual modifications.
Evaluating perceptual quality vectors with RMSE from both human and predicted
labels, we demonstrate that PerMod produces voices with the desired perceptual
qualities for typical voices, but performs poorly on atypical voices.

通过使用 PerMod，一个基于条件隐特征扩散模型的系统，我们可以根据给定的特征向量对声音进行感知修改，以实现期望的感知特性。然而，对于不典型声音，PerMod 的性能较差。

PerMod: 以感知为基础的语音修改与潜在扩散模型

PerMod: Perceptually Grounded Voice Modification with Latent Diffusion  Models

Every sound that we hear is the result of successive convolutional operations
(e.g. room acoustics, microphone characteristics, resonant properties of the
instrument itself, not to mention characteristics and limitations of the sound
reproduction system). In this work we seek to determine the best room in which
to perform a particular piece using AI. Additionally, we use room acoustics as
a way to enhance the perceptual qualities of a given sound. Historically, rooms
(particularly Churches and concert halls) were designed to host and serve
specific musical functions. In some cases the architectural acoustical
qualities enhanced the music performed there. We try to mimic this, as a first
step, by designating room impulse responses that would correlate to producing
enhanced sound quality for particular music. A convolutional architecture is
first trained to take in an audio sample and mimic the ratings of experts with
about 78 % accuracy for various instrument families and notes for perceptual
qualities. This gives us a scoring function for any audio sample which can rate
the perceptual pleasantness of a note automatically. Now, via a library of
about 60,000 synthetic impulse responses mimicking all kinds of room,
materials, etc, we use a simple convolution operation, to transform the sound
as if it was played in a particular room. The perceptual evaluator is used to
rank the musical sounds, and yield the "best room or the concert hall" to play
a sound. As a byproduct it can also use room acoustics to turn a poor quality
sound into a "good" sound.

本研究旨在利用人工智能确定最佳演出场所并利用房间声学来提高声音的知觉质量。通过使用卷积神经网络架构，确定与具体音乐类型和音符相对应的房间脉冲响应，从而为任何音频样本提供感知愉悦度自动评分功能。利用模拟的房间脉冲响应库，将声音转换为在特定房间内播放的效果。此外，还可以利用房间声学将低质量的声音转换为 “好” 的声音。