Audio-only-based wake word spotting (WWS) is challenging under noisy
conditions due to environmental interference in signal transmission. In this
paper, we investigate on designing a compact audio-visual WWS system by
utilizing visual information to alleviate the degradation. Specifically, in
order to use visual information, we first encode the detected lips to
fixed-size vectors with MobileNet and concatenate them with acoustic features
followed by the fusion network for WWS. However, the audio-visual model based
on neural networks requires a large footprint and a high computational
complexity. To meet the application requirements, we introduce a neural network
pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning
manner (LTH-IF), to the single-modal and multi-modal models, respectively.
Tested on our in-house corpus for audio-visual WWS in a home TV scene, the
proposed audio-visual system achieves significant performance improvements over
the single-modality (audio-only or video-only) system under different noisy
conditions. Moreover, LTH-IF pruning can largely reduce the network parameters
and computations with no degradation of WWS performance, leading to a potential
product solution for the TV wake-up scenario.

本文提出了一种使用神经网络剪枝策略的紧凑型音视频唤醒词识别系统，该系统利用 MobileNet 对唇部信息进行编码，并与声学特征进行融合，大大提高了在不同噪声条件下的唤醒词识别性能，可望在电视开机场景下实现实际应用。

基于神经网络剪枝的迭代微调设计紧凑型视听唤醒词识别系统研究

A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

Wake word (WW) spotting is challenging in far-field not only because of the
interference in signal transmission but also the complexity in acoustic
environments. Traditional WW model training requires large amount of in-domain
WW-specific data with substantial human annotations therefore it is hard to
build WW models without such data. In this paper we present data-efficient
solutions to address the challenges in WW modeling, such as domain-mismatch,
noisy conditions, limited annotation, etc. Our proposed system is composed of a
multi-condition training pipeline with a stratified data augmentation, which
improves the model robustness to a variety of predefined acoustic conditions,
together with a semi-supervised learning pipeline to accurately extract the WW
and confusable examples from untranscribed speech corpus. Starting from only 10
hours of domain-mismatched WW audio, we are able to enlarge and enrich the
training dataset by 20-100 times to capture the acoustic complexity. Our
experiments on real user data show that the proposed solutions can achieve
comparable performance of a production-grade model by saving 97\% of the amount
of WW-specific data collection and 86\% of the bandwidth for annotation.

本研究提供了一种数据高效的跨领域唤醒词模型训练技术，该技术包括多种声学环境的训练管道和半监督学习管道，可以用 10 个小时的领域不匹配的音频来增强模型鲁棒性，并从未经译写的语音语料库中准确提取 WW 和可能相似的例子，所提出的解决方案可节省 97％的具体 WW 数据收集和 86％的注释带宽并达到产业级模型的可比性能。