We propose a novel cascaded cross-modal transformer (CCMT) that combines
speech and text transcripts to detect customer requests and complaints in phone
conversations. Our approach leverages a multimodal paradigm by transcribing the
speech using automatic speech recognition (ASR) models and translating the
transcripts into different languages. Subsequently, we combine
language-specific BERT-based models with Wav2Vec2.0 audio features in a novel
cascaded cross-attention transformer model. We apply our system to the Requests
Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics
Challenge, reaching unweighted average recalls (UAR) of 65.41% and 85.87% for
the complaint and request classes, respectively.

我们提出了一种新颖的级联跨模态变换器（CCMT），结合语音和文本转录以检测电话对话中的客户要求和投诉。我们的方法利用多模态范例，通过使用自动语音识别（ASR）模型转录语音并将转录成不同语言。随后，我们将语言特定的基于 BERT 的模型与 Wav2Vec2.0 音频特征结合使用在一个新颖的级联交叉注意力变换器模型中。我们将我们的系统应用到 ACM Multimedia 2023 计算言语学挑战赛的请求子挑战中，分别对于投诉和请求类别达到了 65.41％和 85.87％的非加权平均召回率（UAR）。