Neural speech separation has made remarkable progress and its integration
with automatic speech recognition (ASR) is an important direction towards
realizing multi-speaker ASR. This work provides an insightful investigation of
speech separation in reverberant and noisy-reverberant scenarios as an ASR
front-end. In detail, we explore multi-channel separation methods, mask-based
beamforming and complex spectral mapping, as well as the best features to use
in the ASR back-end model. We employ the recent self-supervised learning
representation (SSLR) as a feature and improve the recognition performance from
the case with filterbank features. To further improve multi-speaker recognition
performance, we present a carefully designed training strategy for integrating
speech separation and recognition with SSLR. The proposed integration using
TF-GridNet-based complex spectral mapping and WavLM-based SSLR achieves a 2.5%
word error rate in reverberant WHAMR! test set, significantly outperforming an
existing mask-based MVDR beamforming and filterbank integration (28.9%).

深度学习在语音分离方面的应用及其对多说话人语音识别的整合进行了深入研究，并提出了一种通过复杂谱映射和自监督学习表示进行的训练策略，从而在噪声和混响环境下显著提升多说话人语音识别性能。

探索自监督学习表征中的语音分离与识别集成

Exploring the Integration of Speech Separation and Recognition with  Self-Supervised Learning Representation

Speaker diarization is a task to label an audio or video recording with the
identity of the speaker at each given time stamp. In this work, we propose a
novel machine learning framework to conduct real-time multi-speaker diarization
and recognition without prior registration and pretraining in a fully online
and reinforcement learning setting. Our framework combines embedding
extraction, clustering, and resegmentation into the same problem as an online
decision-making problem. We discuss practical considerations and advanced
techniques such as the offline reinforcement learning, semi-supervision, and
domain adaptation to address the challenges of limited training data and
out-of-distribution environments. Our approach considers speaker diarization as
a fully online learning problem of the speaker recognition task, where the
agent receives no pretraining from any training set before deployment, and
learns to detect speaker identity on the fly through reward feedbacks. The
paradigm of the reinforcement learning approach to speaker diarization presents
an adaptive, lightweight, and generalizable system that is useful for
multi-user teleconferences, where many people might come and go without
extensive pre-registration ahead of time. Lastly, we provide a desktop
application that uses our proposed approach as a proof of concept. To the best
of our knowledge, this is the first approach to apply a reinforcement learning
approach to the speaker diarization task.

该研究提出了一种基于强化学习的机器学习框架，用于实时的多扬声器识别和分割，并能应对有限的训练数据和分布环境的挑战。