Speech or text representation generated by pre-trained models contains
modal-specific information that could be combined for benefiting spoken
language understanding (SLU) tasks. In this work, we propose a novel
pre-training paradigm termed Continuous Integrate-and-Fire Pre-Training
(CIF-PT). It relies on a simple but effective frame-to-token alignment:
continuous integrate-and-fire (CIF) to bridge the representations between
speech and text. It jointly performs speech-to-text training and language model
distillation through CIF as the pre-training (PT). Evaluated on SLU benchmark
SLURP dataset, CIF-PT outperforms the state-of-the-art model by 1.94% of
accuracy and 2.71% of SLU-F1 on the tasks of intent classification and slot
filling, respectively. We also observe the cross-modal representation extracted
by CIF-PT obtains better performance than other neural interfaces for the tasks
of SLU, including the dominant speech representation learned from
self-supervised pre-training.

提出了一种名为 CIF-PT 的新的预训练范式，通过连续整合与火预训练 (CIF) 来弥合语音和文本之间的表示，其在语音理解任务中的表现优于最新的模型，交叉模态表示的性能也更好。