Acoustic matching aims to re-synthesize an audio clip to sound as if it were
recorded in a target acoustic environment. Existing methods assume access to
paired training data, where the audio is observed in both source and target
environments, but this limits the diversity of training data or requires the
use of simulated data or heuristics to create paired samples. We propose a
self-supervised approach to visual acoustic matching where training samples
include only the target scene image and audio -- without acoustically
mismatched source audio for reference. Our approach jointly learns to
disentangle room acoustics and re-synthesize audio into the target environment,
via a conditional GAN framework and a novel metric that quantifies the level of
residual acoustic information in the de-biased audio. Training with either
in-the-wild web data or simulated data, we demonstrate it outperforms the
state-of-the-art on multiple challenging datasets and a wide variety of
real-world audio and environments.

通过条件生成对抗网络框架和一种测量去偏音频中残留声学信息水平的新型指标，我们提出了一种自监督的视觉声学匹配方法，能够在不使用不匹配的源音频作为参考的情况下，将音频重构为目标环境中的音频并学习解开房间声学效应，无论是通过野外网络数据还是模拟数据进行训练，我们的方法在多个具有挑战性的数据集和各种真实世界的音频和环境中表现优于现有方法。

自监督视觉声音匹配

Self-Supervised Visual Acoustic Matching

Every sound that we hear is the result of successive convolutional operations
(e.g. room acoustics, microphone characteristics, resonant properties of the
instrument itself, not to mention characteristics and limitations of the sound
reproduction system). In this work we seek to determine the best room in which
to perform a particular piece using AI. Additionally, we use room acoustics as
a way to enhance the perceptual qualities of a given sound. Historically, rooms
(particularly Churches and concert halls) were designed to host and serve
specific musical functions. In some cases the architectural acoustical
qualities enhanced the music performed there. We try to mimic this, as a first
step, by designating room impulse responses that would correlate to producing
enhanced sound quality for particular music. A convolutional architecture is
first trained to take in an audio sample and mimic the ratings of experts with
about 78 % accuracy for various instrument families and notes for perceptual
qualities. This gives us a scoring function for any audio sample which can rate
the perceptual pleasantness of a note automatically. Now, via a library of
about 60,000 synthetic impulse responses mimicking all kinds of room,
materials, etc, we use a simple convolution operation, to transform the sound
as if it was played in a particular room. The perceptual evaluator is used to
rank the musical sounds, and yield the "best room or the concert hall" to play
a sound. As a byproduct it can also use room acoustics to turn a poor quality
sound into a "good" sound.

本研究旨在利用人工智能确定最佳演出场所并利用房间声学来提高声音的知觉质量。通过使用卷积神经网络架构，确定与具体音乐类型和音符相对应的房间脉冲响应，从而为任何音频样本提供感知愉悦度自动评分功能。利用模拟的房间脉冲响应库，将声音转换为在特定房间内播放的效果。此外，还可以利用房间声学将低质量的声音转换为 “好” 的声音。

通过 AI 选择的房间声学来增强音乐听觉体验

Enhancing Audio Perception of Music By AI Picked Room Acoustics

Most generative models of audio directly generate samples in one of two
domains: time or frequency. While sufficient to express any signal, these
representations are inefficient, as they do not utilize existing knowledge of
how sound is generated and perceived. A third approach (vocoders/synthesizers)
successfully incorporates strong domain knowledge of signal processing and
perception, but has been less actively researched due to limited expressivity
and difficulty integrating with modern auto-differentiation-based machine
learning methods. In this paper, we introduce the Differentiable Digital Signal
Processing (DDSP) library, which enables direct integration of classic signal
processing elements with deep learning methods. Focusing on audio synthesis, we
achieve high-fidelity generation without the need for large autoregressive
models or adversarial losses, demonstrating that DDSP enables utilizing strong
inductive biases without losing the expressive power of neural networks.
Further, we show that combining interpretable modules permits manipulation of
each separate model component, with applications such as independent control of
pitch and loudness, realistic extrapolation to pitches not seen during
training, blind dereverberation of room acoustics, transfer of extracted room
acoustics to new environments, and transformation of timbre between disparate
sources. In short, DDSP enables an interpretable and modular approach to
generative modeling, without sacrificing the benefits of deep learning. The
library is publicly available at this https URL and we welcome
further contributions from the community and domain experts.

本文介绍了 Differentiable Digital Signal Processing (DDSP) 库的使用，该库将传统信号处理元素与深度学习方法直接集成，以实现音频合成。作者通过使用内插模块的组合来演示 DDSP 的应用程序，如分离音高和响度的独立控制，可靠地推断未在训练中出现的音高，完全盲的混响消除，将提取出的房间声学转移到新环境以及在不同来源之间转换音色。