In this paper, we propose MM-KWS, a novel approach to user-defined keyword
spotting leveraging multi-modal enrollments of text and speech templates.
Unlike previous methods that focus solely on either text or speech features,
MM-KWS extracts phoneme, text, and speech embeddings from both modalities.
These embeddings are then compared with the query speech embedding to detect
the target keywords. To ensure the applicability of MM-KWS across diverse
languages, we utilize a feature extractor incorporating several multilingual
pre-trained models. Subsequently, we validate its effectiveness on Mandarin and
English tasks. In addition, we have integrated advanced data augmentation tools
for hard case mining to enhance MM-KWS in distinguishing confusable words.
Experimental results on the LibriPhrase and WenetPhrase datasets demonstrate
that MM-KWS outperforms prior methods significantly.

提出了 MM-KWS，一种新颖的用户定义关键词识别方法，利用文本和语音模板的多模态注册。通过从文本和语音中提取音素、文本和语音嵌入，将这些嵌入与查询语音嵌入进行比较，以检测目标关键词。为了确保 MM-KWS 在多种语言环境下的适用性，利用了包含多个多语言预训练模型的特征提取器。此外，还整合了高级数据增强工具，以提高 MM-KWS 在区分混淆单词方面的性能。在 LibriPhrase 和 WenetPhrase 数据集上的实验证明，MM-KWS 明显优于之前的方法。