The recently proposed visually grounded speech model SpeechCLIP is an
innovative framework that bridges speech and text through images via CLIP
without relying on text transcription. On this basis, this paper introduces two
extensions to SpeechCLIP. First, we apply the Continuous Integrate-and-Fire
(CIF) module to replace a fixed number of CLS tokens in the cascaded
architecture. Second, we propose a new hybrid architecture that merges the
cascaded and parallel architectures of SpeechCLIP into a multi-task learning
framework. Our experimental evaluation is performed on the Flickr8k and
SpokenCOCO datasets. The results show that in the speech keyword extraction
task, the CIF-based cascaded SpeechCLIP model outperforms the previous cascaded
SpeechCLIP model using a fixed number of CLS tokens. Furthermore, through our
hybrid architecture, cascaded task learning boosts the performance of the
parallel branch in image-speech retrieval tasks.

通过替换固定数量的 CLS 令牌，基于 Continuous Integrate-and-Fire 模块的级联 SpeechCLIP 模型在语音关键词提取任务中优于之前的级联 SpeechCLIP 模型。此外，通过混合架构，级联任务学习提升了图像 - 语音检索任务中并行分支的性能。

SpeechCLIP+: 自我监督多任务表示学习：用于语音的 CLIP 和语音 - 图像数据

SpeechCLIP+: Self-supervised multi-task representation learning for  speech via CLIP and speech-image data

Data-driven speech processing models usually perform well with a large amount
of text supervision, but collecting transcribed speech data is costly.
Therefore, we propose SpeechCLIP, a novel framework bridging speech and text
through images to enhance speech models without transcriptions. We leverage
state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images
and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior
state-of-the-art on image-speech retrieval and performs zero-shot speech-text
retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP
can directly retrieve semantically related keywords from speech.

本文提出了一种名为 SpeechCLIP 的新框架，通过图像将语音和文本结合起来，从而改善语音模型的性能，无需直接从转录中进行监督。SpeechCLIP 使用先进的预训练 HuBERT 和 CLIP 模型，并通过配对的图像和口头字幕进行对齐，实现了零样本语音 - 文本检索和语音中关联关键字的提取。