End-to-end (E2E) spoken language understanding (SLU) is constrained by the
cost of collecting speech-semantics pairs, especially when label domains
change. Hence, we explore \textit{zero-shot} E2E SLU, which learns E2E SLU
without speech-semantics pairs, instead using only speech-text and
text-semantics pairs. Previous work achieved zero-shot by pseudolabeling all
speech-text transcripts with a natural language understanding (NLU) model
learned on text-semantics corpora. However, this method requires the domains of
speech-text and text-semantics to match, which often mismatch due to separate
collections. Furthermore, using the entire speech-text corpus from any domains
leads to \textit{imbalance} and \textit{noise} issues. To address these, we
propose \textit{cross-modal selective self-training} (CMSST). CMSST tackles
imbalance by clustering in a joint space of the three modalities (speech, text,
and semantics) and handles label noise with a selection network. We also
introduce two benchmarks for zero-shot E2E SLU, covering matched and found
speech (mismatched) settings. Experiments show that CMSST improves performance
in both two settings, with significantly reduced sample sizes and training
time.

本研究提出了一种跨模态选择性自训练的方法，使得在没有语音 - 语义配对数据的情况下能够有效学习端到端的口语理解模型，并且证明了其在两个零样本端到端口语理解训练集上都能显著提高性能、缩短训练时间。