Automatic recognition of overlapped speech remains a highly challenging task
to date. Motivated by the bimodal nature of human speech perception, this paper
investigates the use of audio-visual technologies for overlapped speech
recognition. Three issues associated with the construction of audio-visual
speech recognition (AVSR) systems are addressed. First, the basic architecture
designs i.e. end-to-end and hybrid of AVSR systems are investigated. Second,
purposefully designed modality fusion gates are used to robustly integrate the
audio and visual features. Third, in contrast to a traditional pipelined
architecture containing explicit speech separation and recognition components,
a streamlined and integrated AVSR system optimized consistently using the
lattice-free MMI (LF-MMI) discriminative criterion is also proposed. The
proposed LF-MMI time-delay neural network (TDNN) system establishes the
state-of-the-art for the LRS2 dataset. Experiments on overlapped speech
simulated from the LRS2 dataset suggest the proposed AVSR system outperformed
the audio only baseline LF-MMI DNN system by up to 29.98\% absolute in word
error rate (WER) reduction, and produced recognition performance comparable to
a more complex pipelined system. Consistent performance improvements of 4.89\%
absolute in WER reduction over the baseline AVSR system using feature fusion
are also obtained.

本研究探讨了利用视听技术识别重叠的语音信息的三个问题，包括基本体系结构设计，模态融合的建议设计门，以及通过优化的统一的方法来建立 AVSR 系统。实验结果表明，该系统在 LRS2 数据集上的性能超过了传统的语音分离和识别组件的流水线架构，可获得相对于仅音频的基线 LF-MMI DNN 系统高达 29.98% 的字错误率（WER）降低，而采用特征融合技术的 AVSR 系统相对于基准系统进一步提高了 4.89% 的 WER 降低。