This paper proposes a universal sound separation (USS) method capable of
handling untrained sampling frequencies (SFs). The USS aims at separating
arbitrary sources of different types and can be the key technique to realize a
source separator that can be universally used as a preprocessor for any
downstream tasks. To realize a universal source separator, there are two
essential properties: universalities with respect to source types and recording
conditions. The former property has been studied in the USS literature, which
has greatly increased the number of source types that can be handled by a
single neural network. However, the latter property (e.g., SF) has received
less attention despite its necessity. Since the SF varies widely depending on
the downstream tasks, the universal source separator must handle a wide variety
of SFs. In this paper, to encompass the two properties, we propose an
SF-independent (SFI) extension of a computationally efficient USS network,
SuDoRM-RF. The proposed network uses our previously proposed SFI convolutional
layers, which can handle various SFs by generating convolutional kernels in
accordance with an input SF. Experiments show that signal resampling can
degrade the USS performance and the proposed method works more consistently
than signal-resampling-based methods for various SFs.

该论文提出了一种能够处理未经训练的采样频率的通用音频分离（USS）方法，旨在分离任意类型的来源，可作为任何后续任务的预处理器的关键技术，并通过实验验证了该方法相对于基于信号重采样的方法在各种采样频率下的性能优越性。

无采样率依赖的通用音频分离

Sampling-Frequency-Independent Universal Sound Separation

Recent years have seen progress beyond domain-specific sound separation for
speech or music towards universal sound separation for arbitrary sounds. Prior
work on universal sound separation has investigated separating a target sound
out of an audio mixture given a text query. Such text-queried sound separation
systems provide a natural and scalable interface for specifying arbitrary
target sounds. However, supervised text-queried sound separation systems
require costly labeled audio-text pairs for training. Moreover, the audio
provided in existing datasets is often recorded in a controlled environment,
causing a considerable generalization gap to noisy audio in the wild. In this
work, we aim to approach text-queried universal sound separation by using only
unlabeled data. We propose to leverage the visual modality as a bridge to learn
the desired audio-textual correspondence. The proposed CLIPSep model first
encodes the input query into a query vector using the contrastive
language-image pretraining (CLIP) model, and the query vector is then used to
condition an audio separation model to separate out the target sound. While the
model is trained on image-audio pairs extracted from unlabeled videos, at test
time we can instead query the model with text inputs in a zero-shot setting,
thanks to the joint language-image embedding learned by the CLIP model.
Further, videos in the wild often contain off-screen sounds and background
noise that may hinder the model from learning the desired audio-textual
correspondence. To address this problem, we further propose an approach called
noise invariant training for training a query-based sound separation model on
noisy data. Experimental results show that the proposed models successfully
learn text-queried universal sound separation using only noisy unlabeled
videos, even achieving competitive performance against a supervised model in
some settings.

该论文提出了一种使用未标记数据的 CLIPSep 模型实现通用声音分离和去噪的方法。

CLIPSep: 无噪声标记视频的文本查询声音分离学习

CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled  Videos

Deep learning approaches have recently achieved impressive performance on
both audio source separation and sound classification. Most audio source
separation approaches focus only on separating sources belonging to a
restricted domain of source classes, such as speech and music. However, recent
work has demonstrated the possibility of "universal sound separation", which
aims to separate acoustic sources from an open domain, regardless of their
class. In this paper, we utilize the semantic information learned by sound
classifier networks trained on a vast amount of diverse sounds to improve
universal sound separation. In particular, we show that semantic embeddings
extracted from a sound classifier can be used to condition a separation
network, providing it with useful additional information. This approach is
especially useful in an iterative setup, where source estimates from an initial
separation stage and their corresponding classifier-derived embeddings are fed
to a second separation network. By performing a thorough hyperparameter search
consisting of over a thousand experiments, we find that classifier embeddings
from clean sources provide nearly one dB of SNR gain, and our best iterative
models achieve a significant fraction of this oracle performance, establishing
a new state-of-the-art for universal sound separation.

本文研究基于深度学习、语义嵌入和分类网络应用于通用声源分离，并成功建立了新的最先进的模型，从而提高了声源分离的性能。

使用声音分类改进通用声音分离

Improving Universal Sound Separation Using Sound Classification

Recent deep learning approaches have achieved impressive performance on
speech enhancement and separation tasks. However, these approaches have not
been investigated for separating mixtures of arbitrary sounds of different
types, a task we refer to as universal sound separation, and it is unknown how
performance on speech tasks carries over to non-speech tasks. To study this
question, we develop a dataset of mixtures containing arbitrary sounds, and use
it to investigate the space of mask-based separation architectures, varying
both the overall network architecture and the framewise analysis-synthesis
basis for signal transformations. These network architectures include
convolutional long short-term memory networks and time-dilated convolution
stacks inspired by the recent success of time-domain enhancement networks like
ConvTasNet. For the latter architecture, we also propose novel modifications
that further improve separation performance. In terms of the framewise
analysis-synthesis basis, we explore both a short-time Fourier transform (STFT)
and a learnable basis, as used in ConvTasNet. For both of these bases, we also
examine the effect of window size. In particular, for STFTs, we find that
longer windows (25-50 ms) work best for speech/non-speech separation, while
shorter windows (2.5 ms) work best for arbitrary sounds. For learnable bases,
shorter windows (2.5 ms) work best on all tasks. Surprisingly, for universal
sound separation, STFTs outperform learnable bases. Our best methods produce an
improvement in scale-invariant signal-to-distortion ratio of over 13 dB for
speech/non-speech separation and close to 10 dB for universal sound separation.

该研究使用深度学习模型进行了基于掩蔽的语音信号增强和分离任务的研究，并尝试将其应用到任意类型混音的分离任务中，即通用声音分离。在此过程中，作者比较了不同的分析合成基础和网络结构，其中长短时记忆网络和时延卷积堆栈是采用时间域增强网络（ConvTasNet）的架构，对于后者，作者还提出了一些新的改进方法来进一步提高分离性能。最后，作者的研究表明，短时傅立叶变换（STFT）在通用声音分离方面表现优异，而在语音 / 非语音分离方面，长窗口的 STFT（25-50 毫秒）效果明显好于短窗口（2.5 毫秒），对于可学习的基础来说，短窗口（2.5 毫秒）一直是最佳选择。作者的最佳 方法在语音 / 非语音分离和通用声音分离方面都取得了显著的信号失真比的提高。