Spoken Language Understanding (SLU) is a task that aims to extract semantic
information from spoken utterances. Previous research has made progress in
end-to-end SLU by using paired speech-text data, such as pre-trained Automatic
Speech Recognition (ASR) models or paired text as intermediate targets.
However, acquiring paired transcripts is expensive and impractical for
unwritten languages. On the other hand, Textless SLU extracts semantic
information from speech without utilizing paired transcripts. However, the
absence of intermediate targets and training guidance for textless SLU often
results in suboptimal performance. In this work, inspired by the
content-disentangled discrete units from self-supervised speech models, we
proposed to use discrete units as intermediate guidance to improve textless SLU
performance. Our method surpasses the baseline method on five SLU benchmark
corpora. Additionally, we find that unit guidance facilitates few-shot learning
and enhances the model's ability to handle noise.

本研究提出使用内容分离的离散单元作为中间指导，以提高文本缺失情况下的口语理解性能，结果表明该方法在五个 SLU 基准语料库上优于基准方法，并且该单位指导可以促进少样本学习，增强模型处理噪声的能力。

利用离散单元作为中间目标提高无文本口语理解

Improving Textless Spoken Language Understanding with Discrete Units as  Intermediate Target

Direct speech-to-speech translation (S2ST) with discrete units leverages
recent progress in speech representation learning. Specifically, a sequence of
discrete representations derived in a self-supervised manner are predicted from
the model and passed to a vocoder for speech reconstruction, while still facing
the following challenges: 1) Acoustic multimodality: the discrete units derived
from speech with same content could be indeterministic due to the acoustic
property (e.g., rhythm, pitch, and energy), which causes deterioration of
translation accuracy; 2) high latency: current S2ST systems utilize
autoregressive models which predict each unit conditioned on the sequence
previously generated, failing to take full advantage of parallelism. In this
work, we propose TranSpeech, a speech-to-speech translation model with
bilateral perturbation. To alleviate the acoustic multimodal problem, we
propose bilateral perturbation (BiP), which consists of the style normalization
and information enhancement stages, to learn only the linguistic information
from speech samples and generate more deterministic representations. With
reduced multimodality, we step forward and become the first to establish a
non-autoregressive S2ST technique, which repeatedly masks and predicts unit
choices and produces high-accuracy results in just a few cycles. Experimental
results on three language pairs demonstrate that BiP yields an improvement of
2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our
parallel decoding shows a significant reduction of inference latency, enabling
speedup up to 21.4x than autoregressive technique. Audio samples are available
at https://TranSpeech.github.io/

本文提出了使用双向扰动的语音到语音翻译模型 TranSpeech，该模型通过样本的风格归一化和信息提升改善了离散单元之间的声学多模性，并且使用了非自回归技术来提高翻译速度和准确性。实验结果表明，与基准 S2ST 模型相比，BiP 平均提高了 2.9 BLEU 分数，并且我们的并行解码技术使推理延迟大幅降低。