Multimodal intent recognition is a significant task for understanding human
language in real-world multimodal scenes. Most existing intent recognition
methods have limitations in leveraging the multimodal information due to the
restrictions of the benchmark datasets with only text information. This paper
introduces a novel dataset for multimodal intent recognition (MIntRec) to
address this issue. It formulates coarse-grained and fine-grained intent
taxonomies based on the data collected from the TV series Superstore. The
dataset consists of 2,224 high-quality samples with text, video, and audio
modalities and has multimodal annotations among twenty intent categories.
Furthermore, we provide annotated bounding boxes of speakers in each video
segment and achieve an automatic process for speaker annotation. MIntRec is
helpful for researchers to mine relationships between different modalities to
enhance the capability of intent recognition. We extract features from each
modality and model cross-modal interactions by adapting three powerful
multimodal fusion methods to build baselines. Extensive experiments show that
employing the non-verbal modalities achieves substantial improvements compared
with the text-only modality, demonstrating the effectiveness of using
multimodal information for intent recognition. The gap between the
best-performing methods and humans indicates the challenge and importance of
this task for the community. The full dataset and codes are available for use
at this https URL.

这篇论文介绍了一个新的数据集（MIntRec）来解决现有意图识别方法在实现多模态信息利用方面的限制。使用该数据集，通过提取每种模态的特征并采用三种强大的多模态融合方法，可以显著提高意图识别的效果。