Speech classification tasks often require powerful language understanding
models to grasp useful features, which becomes problematic when limited
training data is available. To attain superior classification performance, we
propose to harness the inherent value of multimodal representations by
transcribing speech using automatic speech recognition (ASR) models and
translating the transcripts into different languages via pretrained translation
models. We thus obtain an audio-textual (multimodal) representation for each
data sample. Subsequently, we combine language-specific Bidirectional Encoder
Representations from Transformers (BERT) with Wav2Vec2.0 audio features via a
novel cascaded cross-modal transformer (CCMT). Our model is based on two
cascaded transformer blocks. The first one combines text-specific features from
distinct languages, while the second one combines acoustic features with
multilingual features previously learned by the first transformer block. We
employed our system in the Requests Sub-Challenge of the ACM Multimedia 2023
Computational Paralinguistics Challenge. CCMT was declared the winning
solution, obtaining an unweighted average recall (UAR) of 65.41% and 85.87% for
complaint and request detection, respectively. Moreover, we applied our
framework on the Speech Commands v2 and HarperValleyBank dialog data sets,
surpassing previous studies reporting results on these benchmarks. Our code is
freely available for download at: this https URL

利用多模态表示，通过自动语音识别模型转录语音并通过预训练翻译模型将转录文本翻译成不同语言，结合文本和音频特征使用级联交叉模态变压器 (CCMT) 模型来实现语音分类任务，该模型在 ACM Multimedia 2023 计算语音学挑战中获得不错的性能，并在 Speech Commands v2 和 HarperValleyBank 对话数据集上超过了之前的研究成果。

音频文本分类的级联交叉模态 Transformer

Cascaded Cross-Modal Transformer for Audio-Textual Classification

We propose a novel cascaded cross-modal transformer (CCMT) that combines
speech and text transcripts to detect customer requests and complaints in phone
conversations. Our approach leverages a multimodal paradigm by transcribing the
speech using automatic speech recognition (ASR) models and translating the
transcripts into different languages. Subsequently, we combine
language-specific BERT-based models with Wav2Vec2.0 audio features in a novel
cascaded cross-attention transformer model. We apply our system to the Requests
Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics
Challenge, reaching unweighted average recalls (UAR) of 65.41% and 85.87% for
the complaint and request classes, respectively.

我们提出了一种新颖的级联跨模态变换器（CCMT），结合语音和文本转录以检测电话对话中的客户要求和投诉。我们的方法利用多模态范例，通过使用自动语音识别（ASR）模型转录语音并将转录成不同语言。随后，我们将语言特定的基于 BERT 的模型与 Wav2Vec2.0 音频特征结合使用在一个新颖的级联交叉注意力变换器模型中。我们将我们的系统应用到 ACM Multimedia 2023 计算言语学挑战赛的请求子挑战中，分别对于投诉和请求类别达到了 65.41％和 85.87％的非加权平均召回率（UAR）。