This paper presents an audio-visual approach for voice separation which
produces state-of-the-art results at a low latency in two scenarios: speech and
singing voice. The model is based on a two-stage network. Motion cues are
obtained with a lightweight graph convolutional network that processes face
landmarks. Then, both audio and motion features are fed to an audio-visual
transformer which produces a fairly good estimation of the isolated target
source. In a second stage, the predominant voice is enhanced with an audio-only
network. We present different ablation studies and comparison to
state-of-the-art methods. Finally, we explore the transferability of models
trained for speech separation in the task of singing voice separation. The
demos, code, and weights are available in this https URL

本文提出了一种音频 - 视觉声音分离方案，在两种不同场景（语音和唱歌）中实现了低时延的最新成果。该模型基于两级网络，采用轻量级图卷积网络从面部标记中提取运动线索，然后将视觉和音频特征输入到音频 - 视觉转换器中，为目标源的隔离估计提供相当不错的结果。在第二阶段，利用音频网络增强了主要的声音。我们进行了不同的消融研究和与最先进的方法比较。最后，我们探讨了在唱声分离任务中训练语音分离模型的可转移性。

VoViT: 基于图论的低延迟音视频语音分离 Transformer

VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer

In the recent years, singing voice separation systems showed increased
performance due to the use of supervised training. The design of training
datasets is known as a crucial factor in the performance of such systems. We
investigate on how the characteristics of the training dataset impacts the
separation performances of state-of-the-art singing voice separation
algorithms. We show that the separation quality and diversity are two important
and complementary assets of a good training dataset. We also provide insights
on possible transforms to perform data augmentation for this task.

本研究探讨了训练数据集的特征对半监督唱声分离系统性能的影响和数据增强方法。结果表明，训练数据集的分离质量和多样性是性能的关键因素。

关于训练数据的歌声分离研究

Singing voice separation: a study on training data

Singing voice separation based on deep learning relies on the usage of
time-frequency masking. In many cases the masking process is not a learnable
function or is not encapsulated into the deep learning optimization.
Consequently, most of the existing methods rely on a post processing step using
the generalized Wiener filtering. This work proposes a method that learns and
optimizes (during training) a source-dependent mask and does not need the
aforementioned post processing step. We introduce a recurrent inference
algorithm, a sparse transformation step to improve the mask generation process,
and a learned denoising filter. Obtained results show an increase of 0.49 dB
for the signal to distortion ratio and 0.30 dB for the signal to interference
ratio, compared to previous state-of-the-art approaches for monaural singing
voice separation.

该论文提出了一种基于深度学习的歌声分离方法，学习和优化了源依赖性遮罩，不需要使用后处理步骤，并引入了循环推断算法、稀疏变换步骤和学习去噪滤波器，同时也提高了单声道歌声分离的性能。

使用跳过 - 滤波连接和循环推理时间频率掩模的单声道歌声分离

Monaural Singing Voice Separation with Skip-Filtering Connections and  Recurrent Inference of Time-Frequency Mask

The state of the art in music source separation employs neural networks
trained in a supervised fashion on multi-track databases to estimate the
sources from a given mixture. With only few datasets available, often extensive
data augmentation is used to combat overfitting. Mixing random tracks, however,
can even reduce separation performance as instruments in real music are
strongly correlated. The key concept in our approach is that source estimates
of an optimal separator should be indistinguishable from real source signals.
Based on this idea, we drive the separator towards outputs deemed as realistic
by discriminator networks that are trained to tell apart real from separator
samples. This way, we can also use unpaired source and mixture recordings
without the drawbacks of creating unrealistic music mixtures. Our framework is
widely applicable as it does not assume a specific network architecture or
number of sources. To our knowledge, this is the first adoption of adversarial
training for music source separation. In a prototype experiment for singing
voice separation, separation performance increases with our approach compared
to purely supervised training.

本文介绍了一种新的音乐源分离算法，使用对抗训练让分离器的输出更加真实，并取得了对于歌声分离的较好效果。